Adversarial scenarios for safety testing of autonomous vehicles

ABSTRACT

Techniques to generate driving scenarios for autonomous vehicles characterize a path in a driving scenario according to metrics such as narrowness and effort. Nodes of the path are assigned a time for action to avoid collision from the node. The generated scenarios may be simulated in a computer.

BACKGROUND

Research and development on Autonomous Vehicles (AV) have increased inrecent years in part due to advances in computer vision and deeplearning. Comprehensive safety testing is crucial to a successfuladoption of AV technology, which remains challenging due to itscomplexity.

Traditional software testing methods utilize formal verificationtechniques, or techniques such as code coverage and fuzzing. However,these techniques are not readily applicable to testing of AVs due to thecomplexity of the AV software that often includes components such asdeep neural networks (DNNs). An alternative is to use computers tosimulate driving scenarios for testing AVs. Custom scenarios and cornercases can be tested in simulation under various conditions that are rareto encounter in reality. Physics-based simulators enable comprehensivetesting of AVs under conditions similar to the real-world, andphoto-realistic simulators enable testing of perception modules (e.g.,based on deep neural networks).

Implementing scenarios for testing AVs in a simulation environment canbe difficult. Conventional practice is to utilize a human expert tospecify the driving scenario. This is time consuming and difficult toscale and may miss many corner cases.

Other methods have been proposed for generating scenarios for safetytesting of AVs, including reinforcement learning (e.g., Koren et al.2018) and optimization methods (e.g., Abeysirigoonawardena et al. 2019).In these approaches a scenario is considered useful for testing if itinvolves a collision or a near collision. These scenarios are ofteninsufficient for testing purposes at least because they do not provideinformation as to whether the accident is avoidable or not.

BRIEF SUMMARY

Brief Description of the Several Views of the Drawings

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 depicts an autonomous vehicle simulation environment 100 inaccordance with one embodiment.

FIG. 2 depicts a computer-implemented process 200 in accordance with oneembodiment.

FIG. 3 depicts driving scenarios 300 in accordance with one embodiment.

FIG. 4 depicts a scenario scoring algorithm 400 in accordance with oneembodiment.

FIG. 5 depicts a driving scenario 500 in accordance with one embodiment.

FIG. 6 depicts a driving model 600 in accordance with one embodiment.

FIG. 7 depicts control policy selection modes 700 in accordance with oneembodiment.

FIG. 8 depicts a parallel processing unit 800 in accordance with oneembodiment.

FIG. 9 depicts a general processing cluster 900 in accordance with oneembodiment.

FIG. 10 depicts a memory partition unit 1000 in accordance with oneembodiment.

FIG. 11 depicts a streaming multiprocessor 1100 in accordance with oneembodiment.

FIG. 12 depicts a processing system 1200 in accordance with oneembodiment.

FIG. 13 depicts an exemplary processing system 1300 in accordance withanother embodiment.

FIG. 14 depicts a graphics processing pipeline 1400 in accordance withone embodiment.

FIG. 15 illustrates an autonomous vehicle control system 1500 inaccordance with one embodiment.

FIG. 16 illustrates an autonomous vehicle control system 1600 inaccordance with one embodiment.

FIG. 17 illustrates a virtual environment 1700 in accordance with oneembodiment.

FIG. 18 illustrates a virtual environment 1800 in accordance with oneembodiment.

FIG. 19 illustrates a method 1900 in accordance with one embodiment.

FIG. 20 illustrates a method 2000 in accordance with one embodiment.

FIG. 21 illustrates a method 2100 in accordance with one embodiment.

FIG. 22 illustrates a process 2200 in accordance with one embodiment.

FIG. 23 illustrates a process 2300 in accordance with one embodiment.

FIG. 24 illustrates a method 2400 in accordance with one embodiment.

FIG. 25 illustrates an example autonomous vehicle 2500 in accordancewith one embodiment.

FIG. 26 illustrates a camera locations 2600 in accordance with oneembodiment.

FIG. 27 illustrates a system architecture 2700 in accordance with oneembodiment.

FIG. 28 illustrates a system 2800 in accordance with one embodiment.

DETAILED DESCRIPTION

Autonomous vehicles undergo rigorous safety and reliability testingbefore being deployed on the road. Real-world testing involve arduousdriving times to encounter scenarios involving a collision or an unsafecondition, which are rare. Formal testing methods are often impracticalto apply to AVs. Simulators provide a scalable alternative to real-worldtesting. Disclosed herein are comprehensive techniques forcharacterizing the safety of driving scenarios based on behaviorpolicies having various levels of difficulty. The techniques areindependent of the driver model and are readily applied to computersimulators and well as to real driving scenarios. The techniques may beutilized to generate driving scenarios that end in an unsafe conditionwith high probability.

The driving scenarios may be characterized based on (1) a multivariatemetric for a number of possible safe driving paths (policies) in thescenarios, (2) a narrowness metric for these paths, (3) an effort metricthat considers actuator values, and (4) a minimum amount of time neededfor action to avoid collision. AV specific metrics can also be used tocharacterize the scenarios. For example, metrics based on the error ofthe perception modules and NVIDIA® Safety Force Field assumptions mayaid in characterizing and selecting/developing scenarios customized testthe deployed AV.

A driving scenario comprises environment initial states of a set ofactors and the driving policies of these actors. The inclusion of thedriving policy for the AV is optional. A driving scenario in oneembodiment comprises the following: (1) a set of descriptors for theenvironment, e.g., the number of lanes, their widths, and the roadcurvature, (2) N vehicles {w_(i)}_(i=1 . . . N) and their states{s_(i)(t₀)}_(i=1 . . . N) at time t₀; the state includes the vehicleposition x_(i), orientation θ_(i), speed v_(i), and size of eachvehicle, (3) a set of driving policies {π_(i)}_(i=1 . . . N) associatedwith each non-AV vehicle, (4) one particular vehicle that is an AV,designated the ‘ego vehicle’ w_(E), and its state at time t₀, s_(E)(t₀).A driving policy π_(E) of the AV may also be included.

A scenario may be classified as unsafe when there is an incident betweenthe AV and other actors. The incident may be a collision or a conditionthat is not safe for any of the involved actors (including the AV). Anunsafe scenario may include a driving policy for the AV.

A sequence indicates a temporal succession of states of all thevehicles. A sequence may be obtained by executing a scenario on asimulator using a specific AV policy or may be derived from real-worlddriving data. During a driving sequence, environment and states ofactors any time t may be used to seed a new scenario.

An unsafe sequence is one wherein the AV (w_(E)) collides with anotheractor (or actors) or comes close to other actors such that the distancebetween them is designated unsafe. The time when w_(E) collides withanother actor (say w_(A)) is indicated as t_(acc) (acc being anabbreviation for accident).

From an unsafe sequence, a critical scenario is determined by analyzingback across a time variable from a time when a collision is detected(t_(acc)) until a time to such that a safe path for w_(E) is identifiedthat avoids any collision. In one embodiment, the trajectories arefrozen (made unchanging) for all the other vehicles other than the AVand vehicles creating the unsafe condition in the unsafe sequence,leading up to t_(acc). The time variable of the scenario is stepped backone step from t_(acc) and the trajectory extrapolated at the same speedafter the accident (assuming the accident did not take pace). Thisestimates positions of all actors after t_(acc) as the actors involvedin the accident typically stop moving after the accident and theaccident might take place at a different time than t_(acc) if the AVpolicy is different.

Safe paths for the AV would conventionally be identified by executingsimulations with different driving policies, a computation-intensive andtime-consuming process. Embodiments of the disclosed new approach beginfrom a defined t₀ (e.g., one time step before acc) and compute a set ofpossible locations of w_(E) in the next time step based on its currentstate (location, speed, steering, and acceleration) and configuredrealistic ranges for steering and acceleration values that can appliedat the current state. Possible locations in the next time step aredetermined by calculating the annulus sector corresponding to all thepossible locations for the center of the vehicle, based on a maximumpossible acceleration (positive or negative) and steering of the AV atits current state. The 2D sector may be discretized with a grid with aspecific spacing between the cells computed based on the size of thevehicle (typically ½ or ¼ the vehicle width/height). The cells whosecenters fall into the annulus are recorded as valid next states and theones that are either off-road or result in a collision with anothervehicle are discarded. For each new valid state, this procedure isrepeated until some configured number of time steps after t_(acc) and atree is built with all the possible trajectories that can be executed byw_(E) (starting at t₀). Finding a safe trajectory is then carried out byfinding a path with all valid nodes from the root of the tree to a leafat the last level of the tree (one level in the tree corresponds to onetime step). If no safe path is found, the algorithm backs up onetimestep from t₀ and repeats the procedure until a critical scenario isfound.

Unsafe scenarios may be characterized based on the possible futuredriving states that may be computed depending on diving choices made bythe AV, and from these states a set of metrics may be computed. Sixdifferent metrics may be utilized to characterize a scenario based onpossible driving policies and their efforts. These metrics are describedin further detail below. They are defined such that a higher valueimplies that the scenario is more challenging. Applying these metrics toscore a scenario provides an assessment of the performance of the AV andprioritizes scenarios based on their difficulty and risk to be appliedto further improve the vehicle's behavioral policy. FIG. 4 provides adepiction of one embodiment of such a scenario scoring algorithm 400.

CriticalTime (tcritical): The minimum time (in seconds) required beforetacc such that the accident can be avoided by an AV. A larget_(critical) value implies that the AV should take an action early toavoid the collision. This is set to zero whenever the base sequence hasno accident.

SafePathInv (1/#p): Inverse of the number of safe paths (#p) availableto the AV until one timestep beyond tacc. When 1/#p is one, the AV isconstrained to follow a single trajectory to stay safe on the road. As1/#p tends to zero, the AV has many ways to avoid the accident.

UnsafePercent (pc %): The percentage of paths leading to a collisionwithin a given scenario, among all the paths that do not lead the AVoff-road, in absence of other vehicles.

AvgEffort (E[es]): The effort to navigate from a parent to a child nodeis determined as a sum of the absolute values of the steering andacceleration controls. To compute the effort needed to navigate a safepath (es), effort values of all the edges in the path are added. Theaverage effort is computed for all the safe paths found in a tree toobtain AvgEffort (E[es]). A large value indicates that, on the average,high-effort navigations are needed to avoid the accident.

MinEffort (min[es]): The minimum effort required to navigate through asafe path.

NarrowInv (1/E[min[cs]]): Measures how narrow a safe path is bycomputing the minimum number of children for each of the nodes in a safepath(s). This is referred to as min[cs], where cs indicates the numberof children of a node along the path. E[min[cs]] is the average of theminimum number of children each safe path; its inverse (referred to asNarrowInv) is one (or close to one) when the AV has only one (or few)options at each node to navigate through a safe path. A narrow pathsuggests that an evasive maneuver requires precision during execution.

Other metrics based on (1) AV system's perception quality and (2)assumptions of accident avoidance system (e.g., Safety Force Field) mayalso be used along with all or a subset of the above metrics tocharacterize scenarios.

The state of an actor may be represented as s=[v θ], where v is thescalar velocity and θ is the angular direction. These parameterscorrespond to the vehicle speed and steering direction, which areprimary factors of vehicle operation that may be adjusted by a driver,or by control systems in an AV. The rate of change of the state may berepresented by a control policy:

$f = {\frac{ds}{dt} = \left\lbrack {a\frac{\upsilon\tan(\delta)}{L}} \right\rbrack}$

where a is the acceleration, δ is the steering angle, and L is thewheelbase (distance between front and rear wheels). The position of anyactor in a scenario may be represented by a vector x=[x_(i) x_(j)] ofCartesian coordinates. The formulation for the ego car and an unsafeactor is derived as shown in FIG. 5 , and this formulation is readilyextensible to multiple actors in a scenario. The square of the Euclideandistance between the unsafe actor (A) and the ego car (E) may then beminimized. To achieve this, the control policy for the unsafe actor maybe applied to generate a trajectory that minimizes an objective functionγ defined as:

γ_(AE) =D ² _(AE) =∥x _(A) −x _(E)∥₂ ²=(x _(iA) −x _(iE))²+(x _(jA) −x_(jE))²

where D is the distance between the actor and ego vehicles and|x_(A)−x_(E)∥₂ ² corresponds to the square of the L2 norm, in linearalgebraic terms.

The objective function may be reduced over time by setting a constrainton the derivative of γ_(AE) such that:

$\frac{{\partial\gamma}{AE}}{\partial t} < 0$

Thus:

$\begin{matrix}{{Equation}1} &  \\{\frac{\partial{\gamma}_{AE}}{\partial t} = {{{\frac{\partial{\gamma}_{AE}}{\partial s_{A}}\frac{{ds}_{A}}{dt}} + {\frac{\partial{\gamma}_{AE}}{\partial s_{E}}\frac{{ds}_{E}}{dt}}} < 0}} & (1)\end{matrix}$

Equation 1 becomes:

(Δx _(i) sin(θ_(A))+Δx _(j) cos(θ_(A)))a _(A)+(Δx _(i) v ² _(A)cos(θ_(A))−Δx _(j) v ² _(A) sin(θ_(A)))tan(δ_(A))/L _(A)<const

which may be further simplified to:

β₁ a _(A)+β₂ tan(δ_(A))<const  Equation 2

The control policy for actor A is characterized by steering angle andacceleration and defines a 2D control space generated by (δ_(A)) anda_(A). In this control space, the constraint described in Equation 2creates a linear half-space where the boundary between the half-spacesis a line. One of the linear half-spaces of the control space representsconditions in which the actors are moving away from each other, which isnot useful for studying potential collision conditions. Thus computationmay focus on the linear half-space representing the motion of the actorstowards each other. Additionally, constraints may be placed on themaximum values of steering angle and acceleration. Combining theseconstraints, the control policy of the unsafe actor may be characterizedby points at the intersection of linear half-spaces that bring theactors closer together.

Three modes for selecting a control policy for the unsafe actor in thisconstrained control space are: (1) mode 1, a policy with maximumacceleration and maximum steering, (2) mode 2, a policy with maximumacceleration and minimum steering, and (3) mode 3, a policy with maximumsteering and minimum acceleration. An example of control policyselection modes 700 is depicted in FIG. 7 . The effort of a policy maybe computed as the sum of square values of acceleration and steeringapplied during scenario simulation. With this definition, mode 1corresponds to maximum effort for creating an unsafe scenario satisfyingthe constraints. A further mode 4 may be defined as the minimum effortmode for creating an unsafe scenario satisfying the constraints.

Once the unsafe scenarios are created, they may be characterized orcategorized based on how difficult it is to avoid simulated accidentswithin the scenario, and thus how useful the scenarios may be intraining an AV for accident avoidance. It may not be helpful to trainthe AV using only safe scenarios, or using unsafe scenarios in whichaccidents are unavoidable or very difficult to avoid. By efficientlyproviding a set of unsafe scenarios, automatically (using metrics, notsubjective human intuition) characterized by difficulty of avoidance,the described techniques may improve AV training quality and time.

FIG. 1 depicts an autonomous vehicle simulation environment 100 in oneembodiment. The autonomous vehicle simulation environment 100 maycomprise a computer simulator 102, driving scenarios 104, behavioralpolicies 106, actors 108, road descriptors 110, a driving pathcomputation 112, a metric computation 114, a perturber 116, constraints118, scoring 120 computations, a prioritizer 122, and a policy modeselector 124. Although scoring and prioritization are depicted in thisembodiment based on results of simulation, other embodiments may performone or both of scoring and prioritization prior to simulation, or mayscore and/or prioritize both before and after simulation.

The policy mode selector 124 selects from among available policy modes126 that each implement a different methodology for controlling anunsafe actor in a scenario. The policy mode selector 124 responds to amode selection 128 to select among configured policy modes 126 to applyas the control policy 130 for an actor. The policy mode selector 124determines a control policy 130 for assigning behavioral policies 106may in turn be used to model the behavior of actors 108 in drivingscenarios.

The actors 108 and a set of road descriptors 110 are applied to modeldriving scenarios 104. The driving scenarios 104, actors 108, and roaddescriptors 110 are subject to a metric computation 114.

The driving scenarios 104 and metric computation 114 results are appliedto a driving path computation 112. Constraints 118 may be applied tolimit the range of driving paths to evaluate. A perturber 116 is appliedto the driving path computation 112 to generate perturbations in thepath computations to increase the likelihood that an unsafe drivingcondition results.

The driving path computation 112 results may be input to a computersimulator 102. Simulation results are subject to scoring 120. Thescenario may be scored based on possible driving policies and the efforteach may expend to avoid unsafe conditions at various points in thesimulated scenario time. The scoring 120 may provide an assessment ofthe performance of an AV. The scored scenarios are prioritized by aprioritizer 122 based on their difficulty and risk. Higher scoringscenarios may be applied to improve the utility of AV behavioralpolicies.

Those of ordinary skill in the art may readily recognize that theautonomous vehicle simulation environment 100 may comprise additional ordifferent elements not depicted and unnecessary to this description. Theautonomous vehicle simulation environment 100 may in one embodimentoperate according to the process depicted in FIG. 2 .

FIG. 2 depicts a computer-implemented process 200 in one embodiment. Thecomputer-implemented process 200 may be used for computer simulation ofa driving scenario for an autonomous vehicle. The actions depictedshould not be presumed to occur in the order presented, unless an actiondepends on the result of a previous action to be carried out. If two ormore actions are not conditioned on one another in some way, one skilledin the art may readily ascertain that they may be carried out inparallel, in a time-division fashion, or in a different order.

At block 202, a driving scenario may be configured with a set of roaddescriptors on which a set of actors operate. Each actor may be assigneda behavioral policy, and at least one actor may be the autonomousvehicle. This action may be carried out, for example, by the computerforming the driving scenarios 104 first introduced in FIG. 1 , whichreceives actors 108 configured with the behavioral policies 106, as wellas the road descriptors 110.

At block 204, a multivariate metric for a number of safe driving pathsin the driving scenario may be derived. This may be carried out, forexample, by the computer performing the metric computation 114 of FIG. 1, which accepts the driving scenarios 104, actors 108, and roaddescriptors 110 as input.

At block 206, a narrowness metric may be assigned to each of the pathsin the driving scenario. At block 208, an effort metric may be assignedto nodes of the paths through the driving scenario. These tasks may alsobe carried out by the computer performing the metric computation 114 ofFIG. 1 based on the driving scenarios 104, actors 108, and roaddescriptors 110. The metric computation 114 results may then be appliedto the driving path computation 112.

At block 210, each node of the driving path may be assigned a minimumamount of time for action by the autonomous vehicle to avoid collisionfrom the node. This may be performed by the computer as part of thedriving path computation 112 introduced in FIG. 1 . The driving pathcomputation 112 is subject to the constraints 118 and purtubationsintroduced in FIG. 1 .

At block 212, the driving scenario may be executed in a computersimulation. This may be performed by the computer simulator 102introduced in FIG. 1 . The output from the computer simulator 102 may besubject to scoring 120 by the computer and a prioritizer 122 algorithmbased on performance (such algorithms are known in the art), which mayact to improve a behavior policy of the autonomous vehicle based on theresults of the computer simulation at block 214.

Again, computer simulation need not be applied in some cases beforescoring and/or prioritization are carried out.

FIG. 3 depicts driving scenarios 300 in accordance with one embodiment.Three driving scenarios with low, medium, and high scores are depicted.Scenario (a) in FIG. 3 is a simple scenario in which the AV may avoidthe vehicle attacking from the left by moving almost anywhere (smallSafePathInv) in the open space in front of it. While the AvgEffort isnot minimal due to the variety of the possible safe paths, the MinEffortis small, suggesting that w_(A) can follow a simple trajectory to solvethe scenario. Scenario (b) is comparatively more challenging with oneattacker blocking the lane in front of the AV and the other attackingthe AV from the back. In this case, the safe driving paths exist bydriving the AV to the left. Because the options are limited,UnsafePercent is higher with respect to scenario (a). The MinEffort ishigher because the safe trajectory is curved. Finally, scenario (c) isthe most challenging scenario among the three, where two attackers aremoving behind and in front of the AV. Most of the paths are unsafe (highUnsafePercent), and the road boundary limits the total number of paths(high SafePathInv). FIG. 3 (d) compares different metrics between thesescenarios, where a larger area indicates a higher score.

FIG. 4 depicts a scenario scoring algorithm 400 in accordance with oneembodiment. Depiction (a) in FIG. 4 shows the quantized region on themap where the AV can travel in the next time step based on maximumallowed steering and acceleration. Depiction (b) is the treerepresentation of cell locations for the AV in different time stepsduring the scenario execution. Cells marked “off road” or “collision”represent the invalid locations for the AV in the scenario.

FIG. 5 depicts a driving scenario 500 in accordance with one embodiment.Position and state of the unsafe actor and the ego car in the scenarioare depicted. The square of the Euclidean distance between the actor andthe ego car may be minimized to identify training scenarios forimproving the behavioral policies of AVs.

FIG. 6 depicts a driving model 600 for dynamic actors in a scenario, inaccordance with one embodiment. An actor stays in a state withprobability p1, or changes lane or speed with probabilities p2 and p3,respectively.

FIG. 7 depicts control policy selection modes 700 in one embodiment. Theattacking policy for actor w_(A) is defined by steering angle δ_(A) andacceleration a_(A). In the 2D space (tan(δ_(A)); a_(A)), Equation 2above defines a valid hemispace. Equation 2 divides the (tan(δ_(A));a_(A)) plane into a valid (green) and not valid (red) regions. Withinthe valid region, different attacking modes can be used to decreasea_(E).

Additionally, constraints are placed on the maximum values of steeringangle and acceleration (blue rectangle in FIG. 7 ). Because any point inspace (tan(δ_(A)); a_(A)) that satisfies the above constraint may beused as a control command for w_(A), four different attacking modes maybe distinguished. The first three—(1) maximum acceleration and steering,or maximum effort; (2) minimum acceleration and maximum steering; (3)minimum steering and maximum acceleration—are obtained by selecting thecorresponding point on the edge of the blue rectangle in FIG. 7 . Thefourth policy (minimum effort) is obtained by selecting the point in thevalid hemispace closest to the origin.

The algorithms and techniques disclosed herein may be executed bycomputing devices utilizing one or more graphic processing unit (GPU)and/or general purpose data processor (e.g., a ‘central processing unitor CPU). For example elements of the autonomous vehicle simulationenvironment 100, and actions of the computer-implemented process 200,may be implemented or executed by such devices. Results may be appliedto update the behavior logic (logic embodying behavioral policies) ofautonomous vehicles. Exemplary architectures will now be described thatmay be configured to carry out the techniques disclosed herein on suchdevices.

The following description may use certain acronyms and abbreviations asfollows:

-   -   “DPC” refers to a “data processing cluster”;    -   “GPC” refers to a “general processing cluster”;    -   “I/O” refers to a “input/output”;    -   “L1 cache” refers to “level one cache”;    -   “L2 cache” refers to “level two cache”;    -   “LSU” refers to a “load/store unit”;    -   “MMU” refers to a “memory management unit”;    -   “MPC” refers to an “M-pipe controller”;    -   “PPU” refers to a “parallel processing unit”;    -   “PROP” refers to a “pre-raster operations unit”;    -   “ROP” refers to a “raster operations”;    -   “SFU” refers to a “special function unit”;    -   “SM” refers to a “streaming multiprocessor”;    -   “Viewport SCC” refers to “viewport scale, cull, and clip”;    -   “WDX” refers to a “work distribution crossbar”; and    -   “XBar” refers to a “crossbar”.

Parallel Processing Unit

FIG. 8 depicts a parallel processing unit 800, in accordance with anembodiment. In an embodiment, the parallel processing unit 800 is amulti-threaded processor that is implemented on one or more integratedcircuit devices. The parallel processing unit 800 is a latency hidingarchitecture designed to process many threads in parallel. A thread(e.g., a thread of execution) is an instantiation of a set ofinstructions configured to be executed by the parallel processing unit800. In an embodiment, the parallel processing unit 800 is a graphicsprocessing unit (GPU) configured to implement a graphics renderingpipeline for processing three-dimensional (3D) graphics data in order togenerate two-dimensional (2D) image data for display on a display devicesuch as a liquid crystal display (LCD) device. In other embodiments, theparallel processing unit 800 may be utilized for performinggeneral-purpose computations. While one exemplary parallel processor isprovided herein for illustrative purposes, it should be strongly notedthat such processor is set forth for illustrative purposes only, andthat any processor may be employed to supplement and/or substitute forthe same.

One or more parallel processing unit 800 modules may be configured toaccelerate thousands of High Performance Computing (HPC), data center,and machine learning applications. The parallel processing unit 800 maybe configured to accelerate numerous deep learning systems andapplications including autonomous vehicle platforms, deep learning,high-accuracy speech, image, and text recognition systems, intelligentvideo analytics, molecular simulations, drug discovery, diseasediagnosis, weather forecasting, big data analytics, astronomy, moleculardynamics simulation, financial modeling, robotics, factory automation,real-time language translation, online search optimizations, andpersonalized user recommendations, and the like.

As shown in FIG. 8 , the parallel processing unit 800 includes an I/Ounit 802, a front-end unit 804, a scheduler unit 808, a workdistribution unit 810, a hub 806, a crossbar 814, one or more generalprocessing cluster 900 modules, and one or more memory partition unit1000 modules. The parallel processing unit 800 may be connected to ahost processor or other parallel processing unit 800 modules via one ormore high-speed NVLink 816 interconnects. The parallel processing unit800 may be connected to a host processor or other peripheral devices viaan interconnect 818. The parallel processing unit 800 may also beconnected to a local memory comprising a number of memory 812 devices.In an embodiment, the local memory may comprise a number of dynamicrandom access memory (DRAM) devices. The DRAM devices may be configuredas a high-bandwidth memory (HBM) subsystem, with multiple DRAM diesstacked within each device. The memory 812 may comprise logic toconfigure the parallel processing unit 800 to carry out aspects of thetechniques disclosed herein.

The NVLink 816 interconnect enables systems to scale and include one ormore parallel processing unit 800 modules combined with one or moreCPUs, supports cache coherence between the parallel processing unit 800modules and CPUs, and CPU mastering. Data and/or commands may betransmitted by the NVLink 816 through the hub 806 to/from other units ofthe parallel processing unit 800 such as one or more copy engines, avideo encoder, a video decoder, a power management unit, etc. (notexplicitly shown). The NVLink 816 is described in more detail inconjunction with FIG. 12 .

The I/O unit 802 is configured to transmit and receive communications(e.g., commands, data, etc.) from a host processor (not shown) over theinterconnect 818. The I/O unit 802 may communicate with the hostprocessor directly via the interconnect 818 or through one or moreintermediate devices such as a memory bridge. In an embodiment, the I/Ounit 802 may communicate with one or more other processors, such as oneor more parallel processing unit 800 modules via the interconnect 818.In an embodiment, the I/O unit 802 implements a Peripheral ComponentInterconnect Express (PCIe) interface for communications over a PCIe busand the interconnect 818 is a PCIe bus. In alternative embodiments, theI/O unit 802 may implement other types of well-known interfaces forcommunicating with external devices.

The I/O unit 802 decodes packets received via the interconnect 818. Inan embodiment, the packets represent commands configured to cause theparallel processing unit 800 to perform various operations. The I/O unit802 transmits the decoded commands to various other units of theparallel processing unit 800 as the commands may specify. For example,some commands may be transmitted to the front-end unit 804. Othercommands may be transmitted to the hub 806 or other units of theparallel processing unit 800 such as one or more copy engines, a videoencoder, a video decoder, a power management unit, etc. (not explicitlyshown). In other words, the I/O unit 802 is configured to routecommunications between and among the various logical units of theparallel processing unit 800.

In an embodiment, a program executed by the host processor encodes acommand stream in a buffer that provides workloads to the parallelprocessing unit 800 for processing. A workload may comprise severalinstructions and data to be processed by those instructions. The bufferis a region in a memory that is accessible (e.g., read/write) by boththe host processor and the parallel processing unit 800. For example,the I/O unit 802 may be configured to access the buffer in a systemmemory connected to the interconnect 818 via memory requests transmittedover the interconnect 818. In an embodiment, the host processor writesthe command stream to the buffer and then transmits a pointer to thestart of the command stream to the parallel processing unit 800. Thefront-end unit 804 receives pointers to one or more command streams. Thefront-end unit 804 manages the one or more streams, reading commandsfrom the streams and forwarding commands to the various units of theparallel processing unit 800.

The front-end unit 804 is coupled to a scheduler unit 808 thatconfigures the various general processing cluster 900 modules to processtasks defined by the one or more streams. The scheduler unit 808 isconfigured to track state information related to the various tasksmanaged by the scheduler unit 808. The state may indicate which generalprocessing cluster 900 a task is assigned to, whether the task is activeor inactive, a priority level associated with the task, and so forth.The scheduler unit 808 manages the execution of a plurality of tasks onthe one or more general processing cluster 900 modules.

The scheduler unit 808 is coupled to a work distribution unit 810 thatis configured to dispatch tasks for execution on the general processingcluster 900 modules. The work distribution unit 810 may track a numberof scheduled tasks received from the scheduler unit 808. In anembodiment, the work distribution unit 810 manages a pending task pooland an active task pool for each of the general processing cluster 900modules. The pending task pool may comprise a number of slots (e.g., 32slots) that contain tasks assigned to be processed by a particulargeneral processing cluster 900. The active task pool may comprise anumber of slots (e.g., 4 slots) for tasks that are actively beingprocessed by the general processing cluster 900 modules. As a generalprocessing cluster 900 finishes the execution of a task, that task isevicted from the active task pool for the general processing cluster 900and one of the other tasks from the pending task pool is selected andscheduled for execution on the general processing cluster 900. If anactive task has been idle on the general processing cluster 900, such aswhile waiting for a data dependency to be resolved, then the active taskmay be evicted from the general processing cluster 900 and returned tothe pending task pool while another task in the pending task pool isselected and scheduled for execution on the general processing cluster900.

The work distribution unit 810 communicates with the one or more generalprocessing cluster 900 modules via crossbar 814. The crossbar 814 is aninterconnect network that couples many of the units of the parallelprocessing unit 800 to other units of the parallel processing unit 800.For example, the crossbar 814 may be configured to couple the workdistribution unit 810 to a particular general processing cluster 900.Although not shown explicitly, one or more other units of the parallelprocessing unit 800 may also be connected to the crossbar 814 via thehub 806.

The tasks are managed by the scheduler unit 808 and dispatched to ageneral processing cluster 900 by the work distribution unit 810. Thegeneral processing cluster 900 is configured to process the task andgenerate results. The results may be consumed by other tasks within thegeneral processing cluster 900, routed to a different general processingcluster 900 via the crossbar 814, or stored in the memory 812. Theresults can be written to the memory 812 via the memory partition unit1000 modules, which implement a memory interface for reading and writingdata to/from the memory 812. The results can be transmitted to anotherparallel processing unit 800 or CPU via the NVLink 816. In anembodiment, the parallel processing unit 800 includes a number U ofmemory partition unit 1000 modules that is equal to the number ofseparate and distinct memory 812 devices coupled to the parallelprocessing unit 800. A memory partition unit 1000 will be described inmore detail below in conjunction with FIG. 10 .

In an embodiment, a host processor executes a driver kernel thatimplements an application programming interface (API) that enables oneor more applications executing on the host processor to scheduleoperations for execution on the parallel processing unit 800. In anembodiment, multiple compute applications are simultaneously executed bythe parallel processing unit 800 and the parallel processing unit 800provides isolation, quality of service (QoS), and independent addressspaces for the multiple compute applications. An application maygenerate instructions (e.g., API calls) that cause the driver kernel togenerate one or more tasks for execution by the parallel processing unit800. The driver kernel outputs tasks to one or more streams beingprocessed by the parallel processing unit 800. Each task may compriseone or more groups of related threads, referred to herein as a warp. Inan embodiment, a warp comprises 32 related threads that may be executedin parallel. Cooperating threads may refer to a plurality of threadsincluding instructions to perform the task and that may exchange datathrough shared memory. Threads and cooperating threads are described inmore detail in conjunction with FIG. 11 .

FIG. 9 depicts a general processing cluster 900 of the parallelprocessing unit 800 of FIG. 8 , in accordance with an embodiment. Asshown in FIG. 9 , each general processing cluster 900 includes a numberof hardware units for processing tasks. In an embodiment, each generalprocessing cluster 900 includes a pipeline manager 902, a pre-rasteroperations unit 904, a raster engine 908, a work distribution crossbar914, a memory management unit 916, and one or more data processingcluster 906. It will be appreciated that the general processing cluster900 of FIG. 9 may include other hardware units in lieu of or in additionto the units shown in FIG. 9 .

In an embodiment, the operation of the general processing cluster 900 iscontrolled by the pipeline manager 902. The pipeline manager 902 managesthe configuration of the one or more data processing cluster 906 modulesfor processing tasks allocated to the general processing cluster 900. Inan embodiment, the pipeline manager 902 may configure at least one ofthe one or more data processing cluster 906 modules to implement atleast a portion of a graphics rendering pipeline. For example, a dataprocessing cluster 906 may be configured to execute a vertex shaderprogram on the programmable streaming multiprocessor 1100. The pipelinemanager 902 may also be configured to route packets received from thework distribution unit 810 to the appropriate logical units within thegeneral processing cluster 900. For example, some packets may be routedto fixed function hardware units in the pre-raster operations unit 904and/or raster engine 908 while other packets may be routed to the dataprocessing cluster 906 modules for processing by the primitive engine912 or the streaming multiprocessor 1100. In an embodiment, the pipelinemanager 902 may configure at least one of the one or more dataprocessing cluster 906 modules to implement a neural network modeland/or a computing pipeline.

The pre-raster operations unit 904 is configured to route data generatedby the raster engine 908 and the data processing cluster 906 modules toa Raster Operations (ROP) unit, described in more detail in conjunctionwith FIG. 10 . The pre-raster operations unit 904 may also be configuredto perform optimizations for color blending, organize pixel data,perform address translations, and the like.

The raster engine 908 includes a number of fixed function hardware unitsconfigured to perform various raster operations. In an embodiment, theraster engine 908 includes a setup engine, a coarse raster engine, aculling engine, a clipping engine, a fine raster engine, and a tilecoalescing engine. The setup engine receives transformed vertices andgenerates plane equations associated with the geometric primitivedefined by the vertices. The plane equations are transmitted to thecoarse raster engine to generate coverage information (e.g., an x, ycoverage mask for a tile) for the primitive. The output of the coarseraster engine is transmitted to the culling engine where fragmentsassociated with the primitive that fail a z-test are culled, andtransmitted to a clipping engine where fragments lying outside a viewingfrustum are clipped. Those fragments that survive clipping and cullingmay be passed to the fine raster engine to generate attributes for thepixel fragments based on the plane equations generated by the setupengine. The output of the raster engine 908 comprises fragments to beprocessed, for example, by a fragment shader implemented within a dataprocessing cluster 906.

Each data processing cluster 906 included in the general processingcluster 900 includes an M-pipe controller 910, a primitive engine 912,and one or more streaming multiprocessor 1100 modules. The M-pipecontroller 910 controls the operation of the data processing cluster906, routing packets received from the pipeline manager 902 to theappropriate units in the data processing cluster 906. For example,packets associated with a vertex may be routed to the primitive engine912, which is configured to fetch vertex attributes associated with thevertex from the memory 812. In contrast, packets associated with ashader program may be transmitted to the streaming multiprocessor 1100.

The streaming multiprocessor 1100 comprises a programmable streamingprocessor that is configured to process tasks represented by a number ofthreads. Each streaming multiprocessor 1100 is multi-threaded andconfigured to execute a plurality of threads (e.g., 32 threads) from aparticular group of threads concurrently. In an embodiment, thestreaming multiprocessor 1100 implements a Single-Instruction,Multiple-Data (SIMD) architecture where each thread in a group ofthreads (e.g., a warp) is configured to process a different set of databased on the same set of instructions. All threads in the group ofthreads execute the same instructions. In another embodiment, thestreaming multiprocessor 1100 implements a Single-Instruction, MultipleThread (SIMT) architecture where each thread in a group of threads isconfigured to process a different set of data based on the same set ofinstructions, but where individual threads in the group of threads areallowed to diverge during execution. In an embodiment, a programcounter, call stack, and execution state is maintained for each warp,enabling concurrency between warps and serial execution within warpswhen threads within the warp diverge. In another embodiment, a programcounter, call stack, and execution state is maintained for eachindividual thread, enabling equal concurrency between all threads,within and between warps. When execution state is maintained for eachindividual thread, threads executing the same instructions may beconverged and executed in parallel for maximum efficiency. The streamingmultiprocessor 1100 will be described in more detail below inconjunction with FIG. 11 .

The memory management unit 916 provides an interface between the generalprocessing cluster 900 and the memory partition unit 1000. The memorymanagement unit 916 may provide translation of virtual addresses intophysical addresses, memory protection, and arbitration of memoryrequests. In an embodiment, the memory management unit 916 provides oneor more translation lookaside buffers (TLBs) for performing translationof virtual addresses into physical addresses in the memory 812.

FIG. 10 depicts a memory partition unit 1000 of the parallel processingunit 800 of FIG. 8 , in accordance with an embodiment. As shown in FIG.10 , the memory partition unit 1000 includes a raster operations unit1002, a level two cache 1004, and a memory interface 1006. The memoryinterface 1006 is coupled to the memory 812. Memory interface 1006 mayimplement 32, 64, 128, 1024-bit data buses, or the like, for high-speeddata transfer. In an embodiment, the parallel processing unit 800incorporates U memory interface 1006 modules, one memory interface 1006per pair of memory partition unit 1000 modules, where each pair ofmemory partition unit 1000 modules is connected to a correspondingmemory 812 device. For example, parallel processing unit 800 may beconnected to up to Y memory 812 devices, such as high bandwidth memorystacks or graphics double-data-rate, version 5, synchronous dynamicrandom access memory, or other types of persistent storage.

In an embodiment, the memory interface 1006 implements an HBM2 memoryinterface and Y equals half U. In an embodiment, the HBM2 memory stacksare located on the same physical package as the parallel processing unit800, providing substantial power and area savings compared withconventional GDDR5 SDRAM systems. In an embodiment, each HBM2 stackincludes four memory dies and Y equals 4, with HBM2 stack including two128-bit channels per die for a total of 8 channels and a data bus widthof 1024 bits.

In an embodiment, the memory 812 supports Single-Error CorrectingDouble-Error Detecting (SECDED) Error Correction Code (ECC) to protectdata. ECC provides higher reliability for compute applications that aresensitive to data corruption. Reliability is especially important inlarge-scale cluster computing environments where parallel processingunit 800 modules process very large datasets and/or run applications forextended periods.

In an embodiment, the parallel processing unit 800 implements amulti-level memory hierarchy. In an embodiment, the memory partitionunit 1000 supports a unified memory to provide a single unified virtualaddress space for CPU and parallel processing unit 800 memory, enablingdata sharing between virtual memory systems. In an embodiment thefrequency of accesses by a parallel processing unit 800 to memorylocated on other processors is traced to ensure that memory pages aremoved to the physical memory of the parallel processing unit 800 that isaccessing the pages more frequently. In an embodiment, the NVLink 816supports address translation services allowing the parallel processingunit 800 to directly access a CPU's page tables and providing fullaccess to CPU memory by the parallel processing unit 800.

In an embodiment, copy engines transfer data between multiple parallelprocessing unit 800 modules or between parallel processing unit 800modules and CPUs. The copy engines can generate page faults foraddresses that are not mapped into the page tables. The memory partitionunit 1000 can then service the page faults, mapping the addresses intothe page table, after which the copy engine can perform the transfer. Ina conventional system, memory is pinned (e.g., non-pageable) formultiple copy engine operations between multiple processors,substantially reducing the available memory. With hardware pagefaulting, addresses can be passed to the copy engines without worryingif the memory pages are resident, and the copy process is transparent.

Data from the memory 812 or other system memory may be fetched by thememory partition unit 1000 and stored in the level two cache 1004, whichis located on-chip and is shared between the various general processingcluster 900 modules. As shown, each memory partition unit 1000 includesa portion of the level two cache 1004 associated with a correspondingmemory 812 device. Lower level caches may then be implemented in variousunits within the general processing cluster 900 modules. For example,each of the streaming multiprocessor 1100 modules may implement an L1cache. The L1 cache is private memory that is dedicated to a particularstreaming multiprocessor 1100. Data from the level two cache 1004 may befetched and stored in each of the L1 caches for processing in thefunctional units of the streaming multiprocessor 1100 modules. The leveltwo cache 1004 is coupled to the memory interface 1006 and the crossbar814.

The raster operations unit 1002 performs graphics raster operationsrelated to pixel color, such as color compression, pixel blending, andthe like. The raster operations unit 1002 also implements depth testingin conjunction with the raster engine 908, receiving a depth for asample location associated with a pixel fragment from the culling engineof the raster engine 908. The depth is tested against a correspondingdepth in a depth buffer for a sample location associated with thefragment. If the fragment passes the depth test for the sample location,then the raster operations unit 1002 updates the depth buffer andtransmits a result of the depth test to the raster engine 908. It willbe appreciated that the number of partition memory partition unit 1000modules may be different than the number of general processing cluster900 modules and, therefore, each raster operations unit 1002 may becoupled to each of the general processing cluster 900 modules. Theraster operations unit 1002 tracks packets received from the differentgeneral processing cluster 900 modules and determines which generalprocessing cluster 900 that a result generated by the raster operationsunit 1002 is routed to through the crossbar 814. Although the rasteroperations unit 1002 is included within the memory partition unit 1000in FIG. 10 , in other embodiment, the raster operations unit 1002 may beoutside of the memory partition unit 1000. For example, the rasteroperations unit 1002 may reside in the general processing cluster 900 oranother unit.

FIG. 11 illustrates the streaming multiprocessor 1100 of FIG. 9 , inaccordance with an embodiment. As shown in FIG. 11 , the streamingmultiprocessor 1100 includes an instruction cache 1102, one or morescheduler unit 1104 modules (e.g., such as scheduler unit 808), aregister file 1108, one or more processing core 1110 modules, one ormore special function unit 1112 modules, one or more load/store unit1114 modules, an interconnect network 1116, and a shared memory/L1 cache1118.

As described above, the work distribution unit 810 dispatches tasks forexecution on the general processing cluster 900 modules of the parallelprocessing unit 800. The tasks are allocated to a particular dataprocessing cluster 906 within a general processing cluster 900 and, ifthe task is associated with a shader program, the task may be allocatedto a streaming multiprocessor 1100. The scheduler unit 808 receives thetasks from the work distribution unit 810 and manages instructionscheduling for one or more thread blocks assigned to the streamingmultiprocessor 1100. The scheduler unit 1104 schedules thread blocks forexecution as warps of parallel threads, where each thread block isallocated at least one warp. In an embodiment, each warp executes 32threads. The scheduler unit 1104 may manage a plurality of differentthread blocks, allocating the warps to the different thread blocks andthen dispatching instructions from the plurality of differentcooperative groups to the various functional units (e.g., core 1110modules, special function unit 1112 modules, and load/store unit 1114modules) during each clock cycle.

Cooperative Groups is a programming model for organizing groups ofcommunicating threads that allows developers to express the granularityat which threads are communicating, enabling the expression of richer,more efficient parallel decompositions. Cooperative launch APIs supportsynchronization amongst thread blocks for the execution of parallelalgorithms. Conventional programming models provide a single, simpleconstruct for synchronizing cooperating threads: a barrier across allthreads of a thread block (e.g., the syncthreads( ) function). However,programmers would often like to define groups of threads at smaller thanthread block granularities and synchronize within the defined groups toenable greater performance, design flexibility, and software reuse inthe form of collective group-wide function interfaces.

Cooperative Groups enables programmers to define groups of threadsexplicitly at sub-block (e.g., as small as a single thread) andmulti-block granularities, and to perform collective operations such assynchronization on the threads in a cooperative group. The programmingmodel supports clean composition across software boundaries, so thatlibraries and utility functions can synchronize safely within theirlocal context without having to make assumptions about convergence.Cooperative Groups primitives enable new patterns of cooperativeparallelism, including producer-consumer parallelism, opportunisticparallelism, and global synchronization across an entire grid of threadblocks.

A dispatch 1106 unit is configured within the scheduler unit 1104 totransmit instructions to one or more of the functional units. In oneembodiment, the scheduler unit 1104 includes two dispatch 1106 unitsthat enable two different instructions from the same warp to bedispatched during each clock cycle. In alternative embodiments, eachscheduler unit 1104 may include a single dispatch 1106 unit oradditional dispatch 1106 units.

Each streaming multiprocessor 1100 includes a register file 1108 thatprovides a set of registers for the functional units of the streamingmultiprocessor 1100. In an embodiment, the register file 1108 is dividedbetween each of the functional units such that each functional unit isallocated a dedicated portion of the register file 1108. In anotherembodiment, the register file 1108 is divided between the differentwarps being executed by the streaming multiprocessor 1100. The registerfile 1108 provides temporary storage for operands connected to the datapaths of the functional units.

Each streaming multiprocessor 1100 comprises L processing core 1110modules. In an embodiment, the streaming multiprocessor 1100 includes alarge number (e.g., 128, etc.) of distinct processing core 1110 modules.Each core 1110 may include a fully-pipelined, single-precision,double-precision, and/or mixed precision processing unit that includes afloating point arithmetic logic unit and an integer arithmetic logicunit. In an embodiment, the floating point arithmetic logic unitsimplement the IEEE 754-2008 standard for floating point arithmetic. Inan embodiment, the core 1110 modules include 64 single-precision(32-bit) floating point cores, 64 integer cores, 32 double-precision(64-bit) floating point cores, and 8 tensor cores.

Tensor cores configured to perform matrix operations, and, in anembodiment, one or more tensor cores are included in the core 1110modules. In particular, the tensor cores are configured to perform deeplearning matrix arithmetic, such as convolution operations for neuralnetwork training and inferencing. In an embodiment, each tensor coreoperates on a 4×4 matrix and performs a matrix multiply and accumulateoperation D=A′B+C, where A, B, C, and D are 4×4 matrices.

In an embodiment, the matrix multiply inputs A and B are 16-bit floatingpoint matrices, while the accumulation matrices C and D may be 16-bitfloating point or 32-bit floating point matrices. Tensor Cores operateon 16-bit floating point input data with 32-bit floating pointaccumulation. The 16-bit floating point multiply requires 64 operationsand results in a full precision product that is then accumulated using32-bit floating point addition with the other intermediate products fora 4×4×4 matrix multiply. In practice, Tensor Cores are used to performmuch larger two-dimensional or higher dimensional matrix operations,built up from these smaller elements. An API, such as CUDA 9 C++ API,exposes specialized matrix load, matrix multiply and accumulate, andmatrix store operations to efficiently use Tensor Cores from a CUDA-C++program. At the CUDA level, the warp-level interface assumes 16×16 sizematrices spanning all 32 threads of the warp.

Each streaming multiprocessor 1100 also comprises M special functionunit 1112 modules that perform special functions (e.g., attributeevaluation, reciprocal square root, and the like). In an embodiment, thespecial function unit 1112 modules may include a tree traversal unitconfigured to traverse a hierarchical tree data structure. In anembodiment, the special function unit 1112 modules may include textureunit configured to perform texture map filtering operations. In anembodiment, the texture units are configured to load texture maps (e.g.,a 2D array of texels) from the memory 812 and sample the texture maps toproduce sampled texture values for use in shader programs executed bythe streaming multiprocessor 1100. In an embodiment, the texture mapsare stored in the shared memory/L1 cache 1118. The texture unitsimplement texture operations such as filtering operations using mip-maps(e.g., texture maps of varying levels of detail). In an embodiment, eachstreaming multiprocessor 1100 includes two texture units.

Each streaming multiprocessor 1100 also comprises N load/store unit 1114modules that implement load and store operations between the sharedmemory/L1 cache 1118 and the register file 1108. Each streamingmultiprocessor 1100 includes an interconnect network 1116 that connectseach of the functional units to the register file 1108 and theload/store unit 1114 to the register file 1108 and shared memory/L1cache 1118. In an embodiment, the interconnect network 1116 is acrossbar that can be configured to connect any of the functional unitsto any of the registers in the register file 1108 and connect theload/store unit 1114 modules to the register file 1108 and memorylocations in shared memory/L1 cache 1118.

The shared memory/L1 cache 1118 is an array of on-chip memory thatallows for data storage and communication between the streamingmultiprocessor 1100 and the primitive engine 912 and between threads inthe streaming multiprocessor 1100. In an embodiment, the sharedmemory/L1 cache 1118 comprises 128 KB of storage capacity and is in thepath from the streaming multiprocessor 1100 to the memory partition unit1000. The shared memory/L1 cache 1118 can be used to cache reads andwrites. One or more of the shared memory/L1 cache 1118, level two cache1004, and memory 812 are backing stores.

Combining data cache and shared memory functionality into a singlememory block provides the best overall performance for both types ofmemory accesses. The capacity is usable as a cache by programs that donot use shared memory. For example, if shared memory is configured touse half of the capacity, texture and load/store operations can use theremaining capacity. Integration within the shared memory/L1 cache 1118enables the shared memory/L1 cache 1118 to function as a high-throughputconduit for streaming data while simultaneously providing high-bandwidthand low-latency access to frequently reused data.

When configured for general purpose parallel computation, a simplerconfiguration can be used compared with graphics processing.Specifically, the fixed function graphics processing units shown in FIG.8 , are bypassed, creating a much simpler programming model. In thegeneral purpose parallel computation configuration, the workdistribution unit 810 assigns and distributes blocks of threads directlyto the data processing cluster 906 modules. The threads in a blockexecute the same program, using a unique thread ID in the calculation toensure each thread generates unique results, using the streamingmultiprocessor 1100 to execute the program and perform calculations,shared memory/L1 cache 1118 to communicate between threads, and theload/store unit 1114 to read and write global memory through the sharedmemory/L1 cache 1118 and the memory partition unit 1000. When configuredfor general purpose parallel computation, the streaming multiprocessor1100 can also write commands that the scheduler unit 808 can use tolaunch new work on the data processing cluster 906 modules.

The parallel processing unit 800 may be included in a desktop computer,a laptop computer, a tablet computer, servers, supercomputers, asmart-phone (e.g., a wireless, hand-held device), personal digitalassistant (PDA), a digital camera, a vehicle, a head mounted display, ahand-held electronic device, and the like. In an embodiment, theparallel processing unit 800 is embodied on a single semiconductorsubstrate. In another embodiment, the parallel processing unit 800 isincluded in a system-on-a-chip (SoC) along with one or more otherdevices such as additional parallel processing unit 800 modules, thememory 812, a reduced instruction set computer (RISC) CPU, a memorymanagement unit (MMU), a digital-to-analog converter (DAC), and thelike.

In an embodiment, the parallel processing unit 800 may be included on agraphics card that includes one or more memory devices. The graphicscard may be configured to interface with a PCIe slot on a motherboard ofa desktop computer. In yet another embodiment, the parallel processingunit 800 may be an integrated graphics processing unit (iGPU) orparallel processor included in the chipset of the motherboard.

Exemplary Computing System

Systems with multiple GPUs and CPUs are used in a variety of industriesas developers expose and leverage more parallelism in applications suchas artificial intelligence computing. High-performance GPU-acceleratedsystems with tens to many thousands of compute nodes are deployed indata centers, research facilities, and supercomputers to solve everlarger problems. As the number of processing devices within thehigh-performance systems increases, the communication and data transfermechanisms need to scale to support the increased bandwidth.

FIG. 12 is a conceptual diagram of a processing system 1200 implementedusing the parallel processing unit 800 of FIG. 8 , in accordance with anembodiment. The processing system 1200 includes a central processingunit 1206, switch 1204, and multiple parallel processing unit 800modules each and respective memory 812 modules. The NVLink 816 provideshigh-speed communication links between each of the parallel processingunit 800 modules. Although a particular number of NVLink 816 andinterconnect 818 connections are illustrated in FIG. 12 , the number ofconnections to each parallel processing unit 800 and the centralprocessing unit 1206 may vary. The switch 1204 interfaces between theinterconnect 818 and the central processing unit 1206. The parallelprocessing unit 800 modules, memory 812 modules, and NVLink 816connections may be situated on a single semiconductor platform to form aparallel processing module 1202. In an embodiment, the switch 1204supports two or more protocols to interface between various differentconnections and/or links.

In another embodiment (not shown), the NVLink 816 provides one or morehigh-speed communication links between each of the parallel processingunit 800 modules and the central processing unit 1206 and the switch1204 interfaces between the interconnect 818 and each of the parallelprocessing unit 800 modules. The parallel processing unit 800 modules,memory 812 modules, and interconnect 818 may be situated on a singlesemiconductor platform to form a parallel processing module 1202. In yetanother embodiment (not shown), the interconnect 818 provides one ormore communication links between each of the parallel processing unit800 modules and the central processing unit 1206 and the switch 1204interfaces between each of the parallel processing unit 800 modulesusing the NVLink 816 to provide one or more high-speed communicationlinks between the parallel processing unit 800 modules. In anotherembodiment (not shown), the NVLink 816 provides one or more high-speedcommunication links between the parallel processing unit 800 modules andthe central processing unit 1206 through the switch 1204. In yet anotherembodiment (not shown), the interconnect 818 provides one or morecommunication links between each of the parallel processing unit 800modules directly. One or more of the NVLink 816 high-speed communicationlinks may be implemented as a physical NVLink interconnect or either anon-chip or on-die interconnect using the same protocol as the NVLink816.

In the context of the present description, a single semiconductorplatform may refer to a sole unitary semiconductor-based integratedcircuit fabricated on a die or chip. It should be noted that the termsingle semiconductor platform may also refer to multi-chip modules withincreased connectivity which simulate on-chip operation and makesubstantial improvements over utilizing a conventional busimplementation. Of course, the various circuits or devices may also besituated separately or in various combinations of semiconductorplatforms per the desires of the user. Alternately, the parallelprocessing module 1202 may be implemented as a circuit board substrateand each of the parallel processing unit 800 modules and/or memory 812modules may be packaged devices. In an embodiment, the centralprocessing unit 1206, switch 1204, and the parallel processing module1202 are situated on a single semiconductor platform.

In an embodiment, the signaling rate of each NVLink 816 is 20 to 25Gigabits/second and each parallel processing unit 800 includes sixNVLink 816 interfaces (as shown in FIG. 12 , five NVLink 816 interfacesare included for each parallel processing unit 800). Each NVLink 816provides a data transfer rate of 25 Gigabytes/second in each direction,with six links providing 300 Gigabytes/second. The NVLink 816 can beused exclusively for PPU-to-PPU communication as shown in FIG. 12 , orsome combination of PPU-to-PPU and PPU-to-CPU, when the centralprocessing unit 1206 also includes one or more NVLink 816 interfaces.

In an embodiment, the NVLink 816 allows direct load/store/atomic accessfrom the central processing unit 1206 to each parallel processing unit800 module's memory 812. In an embodiment, the NVLink 816 supportscoherency operations, allowing data read from the memory 812 modules tobe stored in the cache hierarchy of the central processing unit 1206,reducing cache access latency for the central processing unit 1206. Inan embodiment, the NVLink 816 includes support for Address TranslationServices (ATS), allowing the parallel processing unit 800 to directlyaccess page tables within the central processing unit 1206. One or moreof the NVLink 816 may also be configured to operate in a low-power mode.

FIG. 13 depicts an exemplary processing system 1300 in which the variousarchitecture and/or functionality of the various previous embodimentsmay be implemented. As shown, an exemplary processing system 1300 isprovided including at least one central processing unit 1206 that isconnected to a communications bus 1310. The communication communicationsbus 1310 may be implemented using any suitable protocol, such as PCI(Peripheral Component Interconnect), PCI-Express, AGP (AcceleratedGraphics Port), HyperTransport, or any other bus or point-to-pointcommunication protocol(s). The exemplary processing system 1300 alsoincludes a main memory 1302. Control logic (software) and data arestored in the main memory 1302 which may take the form of random accessmemory (RAM).

The exemplary processing system 1300 also includes input devices 1308,the parallel processing module 1202, and display devices 1306, e.g. aconventional CRT (cathode ray tube), LCD (liquid crystal display), LED(light emitting diode), plasma display or the like. User input may bereceived from the input devices 1308, e.g., keyboard, mouse, touchpad,microphone, and the like. Each of the foregoing modules and/or devicesmay even be situated on a single semiconductor platform to form theexemplary processing system 1300. Alternately, the various modules mayalso be situated separately or in various combinations of semiconductorplatforms per the desires of the user.

Further, the exemplary processing system 1300 may be coupled to anetwork (e.g., a telecommunications network, local area network (LAN),wireless network, wide area network (WAN) such as the Internet,peer-to-peer network, cable network, or the like) through a networkinterface 1304 for communication purposes.

The exemplary processing system 1300 may also include a secondarystorage (not shown). The secondary storage includes, for example, a harddisk drive and/or a removable storage drive, representing a floppy diskdrive, a magnetic tape drive, a compact disk drive, digital versatiledisk (DVD) drive, recording device, universal serial bus (USB) flashmemory. The removable storage drive reads from and/or writes to aremovable storage unit in a well-known manner.

Computer programs, or computer control logic algorithms, may be storedin the main memory 1302 and/or the secondary storage. Such computerprograms, when executed, enable the exemplary processing system 1300 toperform various functions. The main memory 1302, the storage, and/or anyother storage are possible examples of computer-readable media.

The architecture and/or functionality of the various previous figuresmay be implemented in the context of a general computer system, acircuit board system, a game console system dedicated for entertainmentpurposes, an application-specific system, and/or any other desiredsystem. For example, the exemplary processing system 1300 may take theform of a desktop computer, a laptop computer, a tablet computer,servers, supercomputers, a smart-phone (e.g., a wireless, hand-helddevice), personal digital assistant (PDA), a digital camera, a vehicle,a head mounted display, a hand-held electronic device, a mobile phonedevice, a television, workstation, game consoles, embedded system,and/or any other type of logic.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

Graphics Processing Pipeline

FIG. 13 is a conceptual diagram of a graphics processing pipeline 1400implemented by the parallel processing unit 800 of FIG. 8 , inaccordance with an embodiment. In an embodiment, the parallel processingunit 800 comprises a graphics processing unit (GPU). The parallelprocessing unit 800 is configured to receive commands that specifyshader programs for processing graphics data. Graphics data may bedefined as a set of primitives such as points, lines, triangles, quads,triangle strips, and the like. Typically, a primitive includes data thatspecifies a number of vertices for the primitive (e.g., in a model-spacecoordinate system) as well as attributes associated with each vertex ofthe primitive. The parallel processing unit 800 can be configured toprocess the graphics primitives to generate a frame buffer (e.g., pixeldata for each of the pixels of the display).

An application writes model data for a scene (e.g., a collection ofvertices and attributes) to a memory such as a system memory or memory812. The model data defines each of the objects that may be visible on adisplay. The application then makes an API call to the driver kernelthat requests the model data to be rendered and displayed. The driverkernel reads the model data and writes commands to the one or morestreams to perform operations to process the model data. The commandsmay reference different shader programs to be implemented on thestreaming multiprocessor 1100 modules of the parallel processing unit800 including one or more of a vertex shader, hull shader, domainshader, geometry shader, and a pixel shader. For example, one or more ofthe streaming multiprocessor 1100 modules may be configured to execute avertex shader program that processes a number of vertices defined by themodel data. In an embodiment, the different streaming multiprocessor1100 modules may be configured to execute different shader programsconcurrently. For example, a first subset of streaming multiprocessor1100 modules may be configured to execute a vertex shader program whilea second subset of streaming multiprocessor 1100 modules may beconfigured to execute a pixel shader program. The first subset ofstreaming multiprocessor 1100 modules processes vertex data to produceprocessed vertex data and writes the processed vertex data to the leveltwo cache 1004 and/or the memory 812. After the processed vertex data israsterized (e.g., transformed from three-dimensional data intotwo-dimensional data in screen space) to produce fragment data, thesecond subset of streaming multiprocessor 1100 modules executes a pixelshader to produce processed fragment data, which is then blended withother processed fragment data and written to the frame buffer in memory812. The vertex shader program and pixel shader program may executeconcurrently, processing different data from the same scene in apipelined fashion until all of the model data for the scene has beenrendered to the frame buffer. Then, the contents of the frame buffer aretransmitted to a display controller for display on a display device.

The graphics processing pipeline 1400 is an abstract flow diagram of theprocessing steps implemented to generate 2D computer-generated imagesfrom 3D geometry data. As is well-known, pipeline architectures mayperform long latency operations more efficiently by splitting up theoperation into a plurality of stages, where the output of each stage iscoupled to the input of the next successive stage. Thus, the graphicsprocessing pipeline 1400 receives input data 601 that is transmittedfrom one stage to the next stage of the graphics processing pipeline1400 to generate output data 1404. In an embodiment, the graphicsprocessing pipeline 1400 may represent a graphics processing pipelinedefined by the OpenGL® API. As an option, the graphics processingpipeline 1400 may be implemented in the context of the functionality andarchitecture of the previous Figures and/or any subsequent Figure(s).

As shown in FIG. 14 , the graphics processing pipeline 1400 comprises apipeline architecture that includes a number of stages. The stagesinclude, but are not limited to, a data assembly 1406 stage, a vertexshading 1408 stage, a primitive assembly 1410 stage, a geometry shading1412 stage, a viewport SCC 1414 stage, a rasterization 1416 stage, afragment shading 1418 stage, and a raster operations 1420 stage. In anembodiment, the input data 1402 comprises commands that configure theprocessing units to implement the stages of the graphics processingpipeline 1400 and geometric primitives (e.g., points, lines, triangles,quads, triangle strips or fans, etc.) to be processed by the stages. Theoutput data 1404 may comprise pixel data (e.g., color data) that iscopied into a frame buffer or other type of surface data structure in amemory.

The data assembly 1406 stage receives the input data 1402 that specifiesvertex data for high-order surfaces, primitives, or the like. The dataassembly 1406 stage collects the vertex data in a temporary storage orqueue, such as by receiving a command from the host processor thatincludes a pointer to a buffer in memory and reading the vertex datafrom the buffer. The vertex data is then transmitted to the vertexshading 1408 stage for processing.

The vertex shading 1408 stage processes vertex data by performing a setof operations (e.g., a vertex shader or a program) once for each of thevertices. Vertices may be, e.g., specified as a 4-coordinate vector(e.g., <x, y, z, w>) associated with one or more vertex attributes(e.g., color, texture coordinates, surface normal, etc.). The vertexshading 1408 stage may manipulate individual vertex attributes such asposition, color, texture coordinates, and the like. In other words, thevertex shading 1408 stage performs operations on the vertex coordinatesor other vertex attributes associated with a vertex. Such operationscommonly including lighting operations (e.g., modifying color attributesfor a vertex) and transformation operations (e.g., modifying thecoordinate space for a vertex). For example, vertices may be specifiedusing coordinates in an object-coordinate space, which are transformedby multiplying the coordinates by a matrix that translates thecoordinates from the object-coordinate space into a world space or anormalized-device-coordinate (NCD) space. The vertex shading 1408 stagegenerates transformed vertex data that is transmitted to the primitiveassembly 1410 stage.

The primitive assembly 1410 stage collects vertices output by the vertexshading 1408 stage and groups the vertices into geometric primitives forprocessing by the geometry shading 1412 stage. For example, theprimitive assembly 1410 stage may be configured to group every threeconsecutive vertices as a geometric primitive (e.g., a triangle) fortransmission to the geometry shading 1412 stage. In some embodiments,specific vertices may be reused for consecutive geometric primitives(e.g., two consecutive triangles in a triangle strip may share twovertices). The primitive assembly 1410 stage transmits geometricprimitives (e.g., a collection of associated vertices) to the geometryshading 1412 stage.

The geometry shading 1412 stage processes geometric primitives byperforming a set of operations (e.g., a geometry shader or program) onthe geometric primitives. Tessellation operations may generate one ormore geometric primitives from each geometric primitive. In other words,the geometry shading 1412 stage may subdivide each geometric primitiveinto a finer mesh of two or more geometric primitives for processing bythe rest of the graphic s processing pipeline 1400. The geometry shading1412 stage transmits geometric primitives to the viewport SCC 1414stage.

In an embodiment, the graphics processing pipeline 1400 may operatewithin a streaming multiprocessor and the vertex shading 1408 stage, theprimitive assembly 1410 stage, the geometry shading 1412 stage, thefragment shading 1418 stage, and/or hardware/software associatedtherewith, may sequentially perform processing operations. Once thesequential processing operations are complete, in an embodiment, theviewport SCC 1414 stage may utilize the data. In an embodiment,primitive data processed by one or more of the stages in the graphicsprocessing pipeline 1400 may be written to a cache (e.g. L1 cache, avertex cache, etc.). In this case, in an embodiment, the viewport SCC1414 stage may access the data in the cache. In an embodiment, theviewport SCC 1414 stage and the rasterization 1416 stage are implementedas fixed function circuitry.

The viewport SCC 1414 stage performs viewport scaling, culling, andclipping of the geometric primitives. Each surface being rendered to isassociated with an abstract camera position. The camera positionrepresents a location of a viewer looking at the scene and defines aviewing frustum that encloses the objects of the scene. The viewingfrustum may include a viewing plane, a rear plane, and four clippingplanes. Any geometric primitive entirely outside of the viewing frustummay be culled (e.g., discarded) because the geometric primitive will notcontribute to the final rendered scene. Any geometric primitive that ispartially inside the viewing frustum and partially outside the viewingfrustum may be clipped (e.g., transformed into a new geometric primitivethat is enclosed within the viewing frustum. Furthermore, geometricprimitives may each be scaled based on a depth of the viewing frustum.All potentially visible geometric primitives are then transmitted to therasterization 1416 stage.

The rasterization 1416 stage converts the 3D geometric primitives into2D fragments (e.g. capable of being utilized for display, etc.). Therasterization 1416 stage may be configured to utilize the vertices ofthe geometric primitives to setup a set of plane equations from whichvarious attributes can be interpolated. The rasterization 1416 stage mayalso compute a coverage mask for a plurality of pixels that indicateswhether one or more sample locations for the pixel intercept thegeometric primitive. In an embodiment, z-testing may also be performedto determine if the geometric primitive is occluded by other geometricprimitives that have already been rasterized. The rasterization 1416stage generates fragment data (e.g., interpolated vertex attributesassociated with a particular sample location for each covered pixel)that are transmitted to the fragment shading 1418 stage.

The fragment shading 1418 stage processes fragment data by performing aset of operations (e.g., a fragment shader or a program) on each of thefragments. The fragment shading 1418 stage may generate pixel data(e.g., color values) for the fragment such as by performing lightingoperations or sampling texture maps using interpolated texturecoordinates for the fragment. The fragment shading 1418 stage generatespixel data that is transmitted to the raster operations 1420 stage.

The raster operations 1420 stage may perform various operations on thepixel data such as performing alpha tests, stencil tests, and blendingthe pixel data with other pixel data corresponding to other fragmentsassociated with the pixel. When the raster operations 1420 stage hasfinished processing the pixel data (e.g., the output data 1404), thepixel data may be written to a render target such as a frame buffer, acolor buffer, or the like.

It will be appreciated that one or more additional stages may beincluded in the graphics processing pipeline 1400 in addition to or inlieu of one or more of the stages described above. Variousimplementations of the abstract graphics processing pipeline mayimplement different stages. Furthermore, one or more of the stagesdescribed above may be excluded from the graphics processing pipeline insome embodiments (such as the geometry shading 1412 stage). Other typesof graphics processing pipelines are contemplated as being within thescope of the present disclosure. Furthermore, any of the stages of thegraphics processing pipeline 1400 may be implemented by one or morededicated hardware units within a graphics processor such as parallelprocessing unit 800. Other stages of the graphics processing pipeline1400 may be implemented by programmable hardware units such as thestreaming multiprocessor 1100 of the parallel processing unit 800.

The graphics processing pipeline 1400 may be implemented via anapplication executed by a host processor, such as a CPU. In anembodiment, a device driver may implement an application programminginterface (API) that defines various functions that can be utilized byan application in order to generate graphical data for display. Thedevice driver is a software program that includes a plurality ofinstructions that control the operation of the parallel processing unit800. The API provides an abstraction for a programmer that lets aprogrammer utilize specialized graphics hardware, such as the parallelprocessing unit 800, to generate the graphical data without requiringthe programmer to utilize the specific instruction set for the parallelprocessing unit 800. The application may include an API call that isrouted to the device driver for the parallel processing unit 800. Thedevice driver interprets the API call and performs various operations torespond to the API call. In some instances, the device driver mayperform operations by executing instructions on the CPU. In otherinstances, the device driver may perform operations, at least in part,by launching operations on the parallel processing unit 800 utilizing aninput/output interface between the CPU and the parallel processing unit800. In an embodiment, the device driver is configured to implement thegraphics processing pipeline 1400 utilizing the hardware of the parallelprocessing unit 800.

Various programs may be executed within the parallel processing unit 800in order to implement the various stages of the graphics processingpipeline 1400. For example, the device driver may launch a kernel on theparallel processing unit 800 to perform the vertex shading 1408 stage onone streaming multiprocessor 1100 (or multiple streaming multiprocessor1100 modules). The device driver (or the initial kernel executed by theparallel processing unit 800) may also launch other kernels on theparallel processing unit 800 to perform other stages of the graphicsprocessing pipeline 1400, such as the geometry shading 1412 stage andthe fragment shading 1418 stage. In addition, some of the stages of thegraphics processing pipeline 1400 may be implemented on fixed unithardware such as a rasterizer or a data assembler implemented within theparallel processing unit 800. It will be appreciated that results fromone kernel may be processed by one or more intervening fixed functionhardware units before being processed by a subsequent kernel on astreaming multiprocessor 1100.

FIG. 15 and FIG. 16 are block diagrams of an example autonomous vehiclecontrol system 1500, in accordance with some embodiments of the presentdisclosure. The disclosed mechanisms may be utilized to generate and/orimprove behavioral policies for the autonomous vehicle control system1500 and autonomous vehicle control system 1600. It should be understoodthat this and other arrangements described herein are set forth only asexamples. Other arrangements and elements (e.g., machines, interfaces,functions, orders, groupings of functions, etc.) may be used in additionto or instead of those shown, and some elements may be omittedaltogether. Further, many of the elements described herein arefunctional entities that may be implemented as discrete or distributedcomponents or in conjunction with other components, and in any suitablecombination and location. Various functions described herein as beingperformed by entities may be carried out by hardware, firmware, and/orsoftware. For instance, various functions may be carried out by aprocessor executing instructions stored in memory.

The illustration of FIG. 15 may represent a more generalizedillustration of the autonomous vehicle control system 1500 as comparedto the illustration of FIG. 16 . The components, features, and/orfunctionality of the autonomous vehicle 1502 described with respect toFIG. 15 and FIG. 16 may be implemented using the features, components,and/or functionality described in more detail herein with respect toFIG. 25 -FIG. 28 .

The autonomous vehicle control system 1500 may include an autonomousvehicle 1502, one or more network(s) 1504, and a remote control system1506. The autonomous vehicle 1502 may include a drive stack 1508,sensor(s) 1510, and/or vehicle control(s) 1512. The drive stack 1508 mayrepresent an autonomous driving software stack, as described in moredetail herein with respect to FIG. 16 . The sensor(s) 1510 may includeany number of sensors of the autonomous vehicle 1502, including, withreference to FIGs. FIG. 25 -FIG. 28 , global navigation satellite systemsensor(s) (GNSS sensor(s) 2514), RADAR sensor(s) 2528, ultrasonicsensor(s) 2542, LIDAR sensor(s) 2530, inertial measurement unitsensor(s) (IMU sensor(s) 2508), microphone(s) 2540, stereo camera(s)2504, wide-view camera(s) 2506, infrared camera(s) 2502, surroundcamera(s) 2518, long range long-range and/or mid-range camera(s) 2602,and/or other sensor types. The sensor(s) 1510 may generate sensor data(e.g., image data) representing a field(s) of view of the sensor(s)1510.

For example, the sensor data may represent a field of view of each of anumber of cameras of the autonomous vehicle 1502. In some examples, thesensor data may be generated from any number of cameras that may providea representation of substantially 360 degrees around the autonomousvehicle 1502 (e.g., fields of view that extend substantially parallel toa ground plane). In such an example, the fields of view may include aleft side of the autonomous vehicle 1502, a rear of the autonomousvehicle 1502, a front of the autonomous vehicle 1502, and/or a side ofthe autonomous vehicle 1502. The sensor data may further be generated toinclude fields of view above and/or below the autonomous vehicle 1502(e.g., of the ground or driving surface around the autonomous vehicle1502 and/or of the space above the autonomous vehicle 1502). In someexamples, the sensor data may be generated to include blind spots of theautonomous vehicle 1502 (e.g., using wing-mirror mounted camera(s)). Asanother example, the sensor data may be generated from some or all ofthe camera(s) illustrated in FIG. 26 . As such, the sensor datagenerated by the autonomous vehicle 1502 may include sensor data fromany number of sensors without departing from the scope of the presentdisclosure.

With reference to FIG. 15 , an image 1520 may include a representationof sensor data (e.g., image data) generated from a front-facing cameraof the autonomous vehicle 1502. The image 1520 may include a two-way,solid line 1522 divided street 1548, such that the autonomous vehicle1502, when following the rules of the road, may not be allowed to crossthe solid line 1522 to pass a vehicle or object in the lane of theautonomous vehicle 1502. In the image 1520, a van 1524 may be stopped inthe lane of the autonomous vehicle 1502 to unload boxes, so theautonomous vehicle 1502 may have come to a stop a safe distance behindthe van 1524. By following the constraints of the autonomous vehicle1502 (e.g., due to the rules of the road), the autonomous vehicle 1502may, without the features and functionality of the present disclosure,remain stopped behind the van 1524 until the van 1524 moves (or may passcontrol to a human operator, depending on the embodiment). However, inthe current autonomous vehicle control system 1500, the autonomousvehicle 1502 may determine, in response to encountering the situationrepresented in the image 1520, to transfer at least partial control tothe remote control system 1506. In other examples, the determination totransfer the control of the autonomous vehicle 1502 (e.g., to initiate aremote control session) may be made by the remote operator (or otherwisemay be made at the remote control system 1506); by a passenger of theautonomous vehicle 1502 (e.g., using a command or signal, such as avoice command, an input to a user interface element, a selection of aphysical button, etc.); and/or by another actor. For example, sensordata may be analyzed at the remote control system 1506 (and/or byanother system remote from the autonomous vehicle 1502) and may be usedto determine whether a remote control session should be initiated.

Although the situation represented in FIG. 15 includes a van 1524blocking the lane of the autonomous vehicle 1502, this is not intendedto be limiting. For example, any number of situations, scenarios, and/orenvironments, including but not limited to those described herein, maylead to a determination by the autonomous vehicle 1502 to transfer atleast partial control to the remote control system 1506 withoutdeparting from the scope of the present disclosure. In other examples,the determination may be made by the remote control system 1506 to takeover control of the autonomous vehicle 1502. In any examples, properconsent may be obtained from the owner and/or operator of the autonomousvehicle 1502 in order to enable takeover by the remote operator of theremote control system 1506.

In addition to the image 1520, the autonomous vehicle 1502 may alsocapture additional sensor data from additional sensor(s) 1510 of theautonomous vehicle 1502, such as from a side-view camera(s), a rear-viewcamera(s), a surround camera(s), a wing-mirror mounted camera(s), aroof-mounted camera(s), parking camera(s) (e.g., with a field(s) of viewof the ground surface around the autonomous vehicle 1502), LIDARsensor(s), RADAR sensor(s), microphone(s), etc. The sensor datagenerated by the sensor(s) 1510 may be transmitted over the network(s)1504 to the remote control system 1506. In some examples, the sensor(s)1510 may generate the sensor data in a first format (e.g., a raw format)that may be of a first data size. In order to minimize bandwidthrequirements, the sensor data may be encoded in a second format that maybe of a second data size less than the first data size (e.g., todecrease the amount of data being sent over the network(s) 1504).

In addition to the sensor data that may be used to generate arepresentation of the environment of the autonomous vehicle 1502,vehicle state data (e.g., representative of the state of the autonomousvehicle 1502) and/or calibration data (e.g., for calibrating the remotecontrol(s) 1514 according to the vehicle control(s) 1512) may also betransmitted over the network(s) 1504 to the remote control system 1506.For example, the vehicle state data and/or the calibration data may bedetermined using one or more sensor(s) 1510 of the autonomous vehicle1502, such as the steering sensor(s) 740, speed sensor(s) 2536, brakesensor(s), IMU sensor(s) 2508, GNSS sensor(s) 2514, and/or othersensor(s) 1510. The vehicle state data may include wheel angles,steering wheel angle, location, gear (e.g., Park, Reverse, Neutral,Drive (PRND)), tire pressure, speed, velocity, orientation, etc. Thecalibration data may include steering sensitivity, braking sensitivity,acceleration sensitivity, etc. In some examples, the calibration datamay be determined based on a make, model, or type of the autonomousvehicle 1502. This information may be encoded in the calibration data bythe autonomous vehicle 1502 and/or may be determined by the remotecontrol system 1506, such as by accessing one or more data stores (e.g.,after determining identification information for the autonomous vehicle1502).

The sensor data, the vehicle state data, and/or the calibration data maybe received by the remote control system 1506 over the network(s) 1504.The network(s) 1504 may include one or more network types, such ascellular networks (e.g., 5G, 4G, LTE, etc.), Wi-Fi networks (e.g., whereaccessible), low power wide-area networks (LPWANs) (e.g., LoRaWAN,SigFox, etc.), and/or other network types. In some examples, theautonomous vehicle 1502 may include one or more modems and/or one ormore antennas for redundancy and/or for communicating over differentnetwork types depending on network availability.

The remote control system 1506 may include a virtual environmentgenerator 1518, a VR headset 1516, and a remote control(s) 1514. Thevirtual environment generator 1518 may use the sensor data, the vehiclestate data, and/or the calibration data to generate a virtualenvironment that may represent the environment (e.g., the real-world orphysical environment, such as the ground surface, the vehicles, thepeople or animals, the buildings, the objects, etc.) in the field(s) ofview of the sensor(s) 1510 of the autonomous vehicle 1502 (e.g., thecamera(s), the LIDAR sensor(s), the RADAR sensor(s), etc.), as well asrepresent at least a portion of the autonomous vehicle 1502 (e.g., aninterior, an exterior, components, features, displays, instrumentpanels, etc.) and/or controls of the autonomous vehicle 1502 (e.g., avirtual steering wheel, a virtual brake pedal, a virtual gas pedal, avirtual blinker, a virtual HMI display, etc.). In some examples, thevirtual environment may include virtual representations of portions ofthe autonomous vehicle 1502 that may not be visible to a driver orpassenger of the autonomous vehicle 1502 in the real-world environment,such as the wheels at an angle (e.g., corresponding to the angle of thewheels of the autonomous vehicle 1502 in the real-world environment asdetermined by the vehicle state data and/or the calibration data), whichmay be viewable from within a virtual cockpit of the virtual vehicle bymaking one or more other components of the virtual vehicle fullytransparent, semi-transparent (e.g., translucent), or removed from therendering altogether.

The virtual environment may be generated from any number of vantagepoints of a remote operator. As non-limiting examples, the virtualenvironment may be generated from a vantage point within a driver's seatof the virtual vehicle (e.g., as illustrated in FIG. 18 ); from withinanother location within the virtual vehicle; and from a position outsideof the virtual vehicle (e.g., as illustrated in FIG. 17 ), such as ontop of the virtual vehicle, to the side of the virtual vehicle, behindthe virtual vehicle, above the virtual vehicle, etc. In some examples,the remote operator may be able to select from any number of differentvantage points and/or may be able to transition between differentvantage points, even in the same remote control session. For example,the remote operator may start a remote control session from a firstvantage point inside the cockpit of the virtual vehicle (e.g., in thedriver's seat), and then, when navigating through a tight space oraround an obstacle, may transition to a second vantage point outside ofthe virtual vehicle where the relationship between the tight space orthe obstacle and the virtual vehicle may be more clearly visualized. Inany example, the desired vantage point of the remote operator may beselectable within the remote control system. The remote operator may beable to set defaults or preferences with respect to vantage points.

The remote operator may be able to set defaults and/or preferences withrespect to other information in the virtual environment, such as therepresentations of information that the remote operator would like tohave available within the virtual environment, or more specifically withrespect to the virtual vehicle in the virtual environment (e.g., theremote operator may select which features of the instrument panel shouldbe populated, what should be displayed on a virtual HMI display, whichportions of the vehicle should be transparent and/or removed, what colorthe virtual vehicle should be, what color the interior should be, etc.).As such, the remote operator may be able to generate a custom version ofthe virtual vehicle within the virtual environment. In any example, evenwhere the virtual vehicle is not the same year, make, model, and/or typeas the autonomous vehicle 1502 in the real-world environment, thevirtual vehicle may be scaled to occupy a substantially similar amountof space in the virtual environment as the autonomous vehicle 1502 inthe real-world environment. As such, even when the virtual vehicle is ofa different size or shape as the vehicle 102, the representation of thevirtual vehicle may provide a more direct visualization to the remoteoperator of the amount of space the autonomous vehicle 1502 occupies inthe real-world environment.

In other examples, the virtual vehicle may be generated according to theyear, make, model, type, and/or other information of the autonomousvehicle 1502 in the real-world environment (e.g., if the autonomousvehicle 1502 is a Year N (e.g. 2019), Make X, and Model Y, the virtualvehicle may represent a vehicle with the dimensions, andsteering/driving profiles consistent with a Year N, Make X, Model Yvehicle). In such examples, the remote operator may still be able tocustomize the virtual vehicle, such as by removing or making transparentcertain features, changing a color, changing an interior design, etc.,but, in some examples, may not be able to customize the general shape orsize of the vehicle.

The virtual environment (e.g., virtual environment 1528) may be renderedand displayed on a display of the VR headset 1516 of the remote operator(e.g., remote operator 1530 located in the driver's seat 1546). Thevirtual environment 1528 may represent a virtual vehicle—that maycorrespond to the autonomous vehicle 1502—from a vantage point of thedriver's seat. The virtual environment 1528 may include a representationof what a passenger of the autonomous vehicle 1502 may see when sittingin the driver's seat. The camera(s) or other sensor(s) 1510 may notcapture the sensor data from the same perspective of a passenger ordriver of the vehicle. As a result, in order to generate the virtualenvironment 1528 (or other virtual environments where the vantage pointdoes not directly correspond to a field(s) of view of the sensor(s)),the sensor data may be manipulated. For example, the sensor data may bedistorted or warped, prior to displaying the rendering on the display ofthe VR headset 1516. In some examples, distorting or warping the sensordata may include performing a fisheye reduction technique on one more ofthe sensor data feeds (e.g., video feeds from one or more camera(s)). Inother examples, distorting or warping the sensor data may includeexecuting a positional warp technique to adjust a vantage point of asensor data feed to a desired vantage point. In such an example, such aswhere a camera(s) is roof-mounted on the autonomous vehicle 1502, apositional warp technique may be used to adjust, or bring down, theimage data feed from roof-level of the camera(s) to eye-level of avirtual driver of the virtual vehicle (e.g., the remote operator).

In examples, the sensor data may be manipulated in order to blend orstitch sensor data corresponding to different fields of view ofdifferent sensors. For example, two or more sensors may be used togenerate the representation of the environment (e.g., a first camerawith a first field of view to the front of the autonomous vehicle 1502,a second camera with a second field of view to a left side of theautonomous vehicle 1502, and so on). In such examples, image or videostitching techniques may be used to stitch together or combine sensordata, such as images or video, to generate a field of view (e.g., 360degrees) for the remote operator with virtually seamless transitionsbetween fields of view represented by the different sensor data fromdifferent sensor(s) 1510. In one or more example embodiments, the sensordata may be manipulated and presented to the remote operator in a 3Dvisualization (e.g., stereoscopically). For example, one or more stereocamera(s) 2504 of the autonomous vehicle 1502 may generate images, andthe images may be used (e.g., using one or more neural networks, usingphotometric consistency, etc.) to determine depth (e.g., along a Z-axis)for portions of the real-world environment that correspond to theimages. As such, the 3D visualization may be generated using thestereoscopic depth information from the stereo camera(s) 2504. In otherexamples, the depth information may be generated using LIDAR sensors,RADAR sensors, and/or other sensors of the autonomous vehicle 1502. Inany example, the depth information may be leveraged to generate the 3Dvisualization for display or presentation to the remote operator withinthe virtual environment. In such examples, some or all of rendering ordisplay of the virtual environment to the remote operator may include a3D visualization.

In some examples, because the autonomous vehicle 1502 may be anautonomous vehicle capable of operating at autonomous driving level 5(e.g., fully autonomous driving), the autonomous vehicle 1502 may notinclude a steering wheel. However, even in such examples, the virtualvehicle may include the steering wheel 1532 (e.g., in a positionrelative to a driver's seat, if the autonomous vehicle 1502 had adriver's seat) in order to provide the remote operator 1530 a naturalpoint of reference for controlling the virtual vehicle. In addition tothe steering wheel 1532, the interior of the virtual vehicle may includea rear-view mirror 1536 (which may be rendered to display image datarepresentative of a field(s) of view of a rear-facing camera(s)), wingmirrors (which may be rendered to display image data representative offield(s) of view of side-view camera(s), wing-mounted camera(s), etc.),a virtual HMI display 1534, door handles, doors, a roof, a sunroof,seats, consoles, and/or other portions of the virtual vehicle (e.g.,based on default settings, based on preferences of the remote operator,and/or preferences of another user(s) of the remote control system 1506,etc.).

As described herein, at least some of the portions of the virtualvehicle may be made at least partially transparent and/or be removedfrom the virtual environment. An example is support column 1538 of thevehicle chassis being at least partially transparent and/or removed fromthe virtual vehicle, such that objects and the surface in the virtualenvironment are not occluded or at least less occluded by the supportcolumn 1538. Examples of the virtual environment 1528 are described inmore detail herein with respect to FIG. 18 .

The instance of the virtual environment 1528 in FIG. 15 (andcorrespondingly, FIG. 18 ) may represent a time that the image 1520 wascaptured by the autonomous vehicle 1502, and thus may include, as viewedthrough a windshield of the virtual vehicle, virtual representations ofthe van 1524, the unloaded boxes 1526, the divided street 1548, and thesolid line 1522. In some examples, such as in the virtual environment1528, each of the virtual objects in the virtual environment may berendered relative to the virtual vehicle to correspond to the relativelocation of the objects in the real-world environment with respect tothe autonomous vehicle 1502 (e.g., using depth information from thesensor data). The virtual representations of the image data may includethe images or video from the image data, rendered within the virtualenvironment. As described herein, the virtual environment may berendered from any of a number of different vantage points (includingthose illustrated in FIG. 17-FIG. 18 ), and the virtual environment 1528is only one, non-limiting example of a virtual environment.

The remote operator 1530 may use the remote control(s) 1514 to controlthe virtual vehicle in the virtual environment. The remote control(s)1514 may include a steering wheel 1540 (or other control(s) forproviding steering inputs, such as keyboards, joysticks, handheldcontrollers, etc.), an acceleration component 1542 (which may be aphysical pedal as illustrated in FIG. 15 , or may be a keyboard, ajoystick, a handheld controller, a button, etc.), a braking component1544 (which may be a physical pedal as illustrated in FIG. 15 , or maybe a keyboard, a joystick, a handheld controller, a button, etc.),and/or other control components, such as blinker actuators (which may bephysical levers, or may be controlled using a keyboard, a joystick, ahandheld controller, voice, etc.), a horn, light actuators (such as abutton, lever, or knob for turning on and off lights, including drivinglights, fog lights, high-beams, etc.), etc.

In some examples, the remote control(s) may include pointers (e.g.,controllers or other objects) that may be used to indicate or identify alocation in the environment that the virtual vehicle should navigate to.In such examples, the remote control(s) 1514 may be used to provideinput to the autonomous vehicle 1502 as to where in the real-worldenvironment the autonomous vehicle 1502 should navigate, and theautonomous vehicle 1502 may use this information to generate controlsfor navigating to the location. For example, with respect to the image1520, the remote operator 1530 may point to a location in the lane tothe left of the autonomous vehicle 1502 and the van 1524, such that theautonomous vehicle 1502 is able to use the information to override therules of the road that have stopped the vehicle from passing the van1524, and to proceed to the adjacent lane in order to pass the van 1524and the unloaded boxes 1526. More detail is provided herein for controlinput types with respect to FIG. 16 .

In any example, the remote operator 1530 may control the virtual vehiclethrough the virtual environment 1528, and the control inputs to theremote control(s) 1514 may be captured. Control data representative ofeach of the control inputs (e.g., as they are received by the remotecontrol system 1506) may be transmitted to the autonomous vehicle 1502over the network(s) 1504. In some examples, as described in more detailherein, the control data may be encoded by the remote control system1506 prior to transmission and/or may be encoded upon receipt by theautonomous vehicle 1502. The encoding may be to convert the control datafrom the remote control system 1506 to vehicle control data suitable foruse by the autonomous vehicle 1502. The control data may be scaled,undergo a format change, and/or other encoding may be executed toconvert the control data to vehicle control data that the autonomousvehicle 1502 understands and can execute. As a result, as the remoteoperator 1530 controls the virtual vehicle through the virtualenvironment, the autonomous vehicle 1502 may be controlled through thereal-world environment accordingly. With respect to the image 1520 andthe virtual environment 1528, the remote operator 1530 may control thevirtual vehicle to navigate around the virtual representation of the van1524 by entering the adjacent lane of the divided street 1548 to theleft of the van 1524, passing the van 1524, and then reentering theoriginal lane. Responsive to the input controls from the remote operator1530, the autonomous vehicle 1502 may, at substantially the same time,navigate around the van 1524 by entering the adjacent lane of thedivided street 1548 in the real-world environment, proceeding past thevan 1524, and then reentering the original lane of the divided street1548.

In some examples, such as depending on the preferences of the ownerand/or operator of the autonomous vehicle 1502, a remote control sessionmay be substantially seamless to any passengers of the vehicle 102, suchthat the passengers may not be made aware or notice the transfer ofcontrol to the remote control system 1506 and then back to theautonomous vehicle 1502. In other examples, further depending on thepreferences of the owner and/or operator, the passengers of the vehiclemay be informed prior to and/or during the time when the control ispassed to the remote control system 1506. For example, the remotecontrol system 1506 may include a microphone(s) and/or a speaker(s)(e.g., headphones, standalone speakers, etc.), and the autonomousvehicle 1502 may include a microphone(s) and/or a speaker(s), such thatone-way or two-way communication may take place between the passengersand the remote operator 1530. In such examples, once control is passedback to the autonomous vehicle 1502, the passengers may again be madeaware of the transition.

Now referring to FIG. 16 , FIG. 16 may include a more detailedillustration of the autonomous vehicle control system 1500 of FIG. 15 .The autonomous vehicle 1502 may include the drive stack 1508, which mayinclude a sensor manager 1602, perception component(s) 1604 (e.g.,corresponding to a perception layer of the drive stack 1508), a worldmodel manager 1606, planning component(s) 1608 (e.g., corresponding to aplanning layer of the drive stack 1508), control component(s) 1610(e.g., corresponding to a control layer of the drive stack 1508),obstacle avoidance component(s) (e.g., corresponding to an obstacle orcollision avoidance layer of the drive stack 1508), actuationcomponent(s) 1614 (e.g., corresponding to an actuation layer of thedrive stack 1508), and/or other components corresponding to additionaland/or alternative layers of the drive stack 1508.

The sensor manager 1602 may manage and/or abstract sensor data fromsensor(s) 1510 of the autonomous vehicle 1502. For example, and withreference to FIG. 27 , the sensor data may be generated (e.g.,perpetually, at intervals, based on certain conditions) by GNSSsensor(s) 2514, RADAR sensor(s) 2528, ultrasonic sensor(s) 2542, LIDARsensor(s) 2530, IMU sensor(s) 2508, microphone(s) 2540, stereo camera(s)2504, wide-view camera(s) 2506, infrared camera(s) 2502, surroundcamera(s) 2518, long range long-range and/or mid-range camera(s) 2602,and/or other sensor types.

The sensor manager 1602 may receive the sensor data from the sensors indifferent formats (e.g., sensors of the same type, such as LIDARsensors, may output sensor data in different formats), and may beconfigured to convert the different formats to a uniform format (e.g.,for each sensor of the same type). As a result, other components,features, and/or functionality of the autonomous vehicle 1502 may usethe uniform format, thereby simplifying processing of the sensor data.In some examples, the sensor manager 1602 may use a uniform format toapply control back to the sensors of the autonomous vehicle 1502, suchas to set frame rates or to perform video gain control. The sensormanager 1602 may also update sensor packets or communicationscorresponding to the sensor data with timestamps to help informprocessing of the sensor data by various components, features, andfunctionality of the autonomous vehicle control system 1500.

A world model manager 1606 may be used to generate, update, and/ordefine a world model. The world model manager 1606 may use informationgenerated by and received from the perception component(s) 1604 of thedrive stack 1508. The perception component(s) 1604 may include anobstacle perceiver, a path perceiver, a wait perceiver, a map perceiver,and/or other perception component(s) 1604. For example, the world modelmay be defined, at least in part, based on affordances for obstacles,paths, and wait conditions that can be perceived in real-time or nearreal-time by the obstacle perceiver, the path perceiver, the waitperceiver, and/or the map perceiver. The world model manager 1606 maycontinually update the world model based on newly generated and/orreceived inputs (e.g., data) from the obstacle perceiver, the pathperceiver, the wait perceiver, the map perceiver, and/or othercomponents of the autonomous vehicle control system 1500.

The world model may be used to help inform planning component(s) 1608,control component(s) 1610, obstacle avoidance component(s) 1612, and/oractuation component(s) 1614 of the drive stack 1508. The obstacleperceiver may perform obstacle perception that may be based on where theautonomous vehicle 1502 is allowed to drive or is capable of driving,and how fast the autonomous vehicle 1502 can drive without collidingwith an obstacle (e.g., an object, such as a structure, entity, vehicle,etc.) that is sensed by the sensor(s) 1510 of the autonomous vehicle1502.

The path perceiver may perform path perception, such as by perceivingnominal paths that are available in a particular situation. In someexamples, the path perceiver may further take into account lane changesfor path perception. A lane graph may represent the path or pathsavailable to the autonomous vehicle 1502, and may be as simple as asingle path on a highway on-ramp. In some examples, the lane graph mayinclude paths to a desired lane and/or may indicate available changesdown the highway (or other road type), or may include nearby lanes, lanechanges, forks, turns, cloverleaf interchanges, merges, and/or otherinformation.

The wait perceiver may be responsible to determining constraints on theautonomous vehicle 1502 as a result of rules, conventions, and/orpractical considerations. For example, the rules, conventions, and/orpractical considerations may be in relation to traffic lights, multi-waystops, yields, merges, toll booths, gates, police or other emergencypersonnel, road workers, stopped busses or other vehicles, one-waybridge arbitrations, ferry entrances, etc. In some examples, the waitperceiver may be responsible for determining longitudinal constraints onthe autonomous vehicle 1502 that require the vehicle to wait or slowdown until some condition is true. In some examples, wait conditionsarise from potential obstacles, such as crossing traffic in anintersection, that may not be perceivable by direct sensing by theobstacle perceiver, for example (e.g., by using sensor data from thesensor(s) 1510, because the obstacles may be occluded from field ofviews of the sensor(s) 1510). As a result, the wait perceiver mayprovide situational awareness by resolving the danger of obstacles thatare not always immediately perceivable through rules and conventionsthat can be perceived and/or learned. Thus, the wait perceiver may beleveraged to identify potential obstacles and implement one or morecontrols (e.g., slowing down, coming to a stop, etc.) that may not havebeen possible relying solely on the obstacle perceiver.

The map perceiver may include a mechanism by which behaviors arediscerned, and in some examples, to determine specific examples of whatconventions are applied at a particular locale. For example, the mapperceiver may determine, from data representing prior drives or trips,that at a certain intersection there are no U-turns between certainhours, that an electronic sign showing directionality of lanes changesdepending on the time of day, that two traffic lights in close proximity(e.g., barely offset from one another) are associated with differentroads, that in Rhode Island, the first car waiting to make a left turnat traffic light breaks the law by turning before oncoming traffic whenthe light turns green, and/or other information. The map perceiver mayinform the autonomous vehicle 1502 of static or stationaryinfrastructure objects and obstacles. The map perceiver may alsogenerate information for the wait perceiver and/or the path perceiver,for example, such as to determine which light at an intersection has tobe green for the autonomous vehicle 1502 to take a particular path.

In some examples, information from the map perceiver may be sent,transmitted, and/or provided to server(s) (e.g., to a map manager ofserver(s) 2802 of FIG. 28 ), and information from the server(s) may besent, transmitted, and/or provided to the map perceiver and/or alocalization manager of the autonomous vehicle 1502. The map manager mayinclude a cloud mapping application that is remotely located from theautonomous vehicle 1502 and accessible by the autonomous vehicle 1502over the network(s) 1504. For example, the map perceiver and/or thelocalization manager of the autonomous vehicle 1502 may communicate withthe map manager and/or one or more other components or features of theserver(s) to inform the map perceiver and/or the localization manager ofpast and present drives or trips of the autonomous vehicle 1502, as wellas past and present drives or trips of other vehicles. The map managermay provide mapping outputs (e.g., map data) that may be localized bythe localization manager based on a particular location of theautonomous vehicle 1502, and the localized mapping outputs may be usedby the world model manager 1606 to generate and/or update the worldmodel.

In any example, when a determination is made, based on information fromthe path perceiver, the wait perceiver, the map perceiver, the obstacleperceiver, and/or another component of the perception component(s) 1604,that prevents the autonomous vehicle 1502 from proceeding through acertain situation, scenario, and/or environment, at least partialcontrol may be transferred to the remote control system 1506. In someexamples, the passengers of the autonomous vehicle 1502 may be given anoption to wait until the autonomous vehicle 1502 is able to proceedbased on internal rules, conventions, standards, constraints, etc., orto transfer the control to the remote control system 1506 to enable theremote operator to navigate the autonomous vehicle 1502 through thesituation, scenario, and/or environment. The remote operator 1530, oncegiven control, may provide control inputs to the remote control(s) 1514,and the autonomous vehicle 1502 may execute vehicle controlscorresponding to the control inputs that are understandable to theautonomous vehicle 1502.

The planning component(s) 1608 may include a route planner, a laneplanner, a behavior planner, and a behavior selector, among othercomponents, features, and/or functionality. The route planner may usethe information from the map perceiver, the map manager, and/or thelocalization manger, among other information, to generate a planned paththat may consist of GNSS waypoints (e.g., GPS waypoints). The waypointsmay be representative of a specific distance into the future for theautonomous vehicle 1502, such as a number of city blocks, a number ofkilometers/miles, a number of meters/feet, etc., that may be used as atarget for the lane planner.

The lane planner may use the lane graph (e.g., the lane graph from thepath perceiver), object poses within the lane graph (e.g., according tothe localization manager), and/or a target point and direction at thedistance into the future from the route planner as inputs. The targetpoint and direction may be mapped to the best matching drivable pointand direction in the lane graph (e.g., based on GNSS and/or compassdirection). A graph search algorithm may then be executed on the lanegraph from a current edge in the lane graph to find the shortest path tothe target point.

The behavior planner may determine the feasibility of basic behaviors ofthe autonomous vehicle 1502, such as staying in the lane or changinglanes left or right, so that the feasible behaviors may be matched upwith the most desired behaviors output from the lane planner. Forexample, if the desired behavior is determined to not be safe and/oravailable, a default behavior may be selected instead (e.g., defaultbehavior may be to stay in lane when desired behavior or changing lanesis not safe).

The control component(s) 1610 may follow a trajectory or path (lateraland longitudinal) that has been received from the behavior selector ofthe planning component(s) 1608 as closely as possible and within thecapabilities of the autonomous vehicle 1502. In some examples, theremote operator may determine the trajectory or path, and may thus takethe place of or augment the behavior selector. In such examples, theremote operator may provide controls that may be received by the controlcomponent(s) 1610, and the control component(s) may follow the controlsdirectly, may follow the controls as closely as possible within thecapabilities of the vehicle, or may take the controls as a suggestionand determine, using one or more layers of the drive stack 1508, whetherthe controls should be executed or whether other controls should beexecuted.

The control component(s) 1610 may use tight feedback to handle unplannedevents or behaviors that are not modeled and/or anything that causesdiscrepancies from the ideal (e.g., unexpected delay). In some examples,the control component(s) 1610 may use a forward prediction model thattakes control as an input variable, and produces predictions that may becompared with the desired state (e.g., compared with the desired lateraland longitudinal path requested by the planning component(s) 1608). Thecontrol(s) that minimize discrepancy may be determined.

Although the planning component(s) 1608 and the control component(s)1610 are illustrated separately, this is not intended to be limiting.For example, in some embodiments, the delineation between the planningcomponent(s) 1608 and the control component(s) 1610 may not be preciselydefined. As such, at least some of the components, features, and/orfunctionality attributed to the planning component(s) 1608 may beassociated with the control component(s) 1610, and vice versa.

The obstacle avoidance component(s) 1612 may aid the autonomous vehicle1502 in avoiding collisions with objects (e.g., moving and stationaryobjects). The obstacle avoidance component(s) 1612 may include acomputational mechanism at a “primal level” of obstacle avoidance thatmay act as a “survival brain” or “reptile brain” for the autonomousvehicle 1502. In some examples, the obstacle avoidance component(s) 1612may be used independently of components, features, and/or functionalityof the autonomous vehicle 1502 that is required to obey traffic rulesand drive courteously. In such examples, the obstacle avoidancecomponent(s) may ignore traffic laws, rules of the road, and courteousdriving norms in order to ensure that collisions do not occur betweenthe autonomous vehicle 1502 and any objects. As such, the obstacleavoidance layer may be a separate layer from the rules of the roadlayer, and the obstacle avoidance layer may ensure that the autonomousvehicle 1502 is only performing safe actions from an obstacle avoidancestandpoint. The rules of the road layer, on the other hand, may ensurethat vehicle obeys traffic laws and conventions, and observes lawful andconventional right of way (as described herein).

In some examples, when controls are received from the remote controlsystem 1506, the obstacle avoidance component(s) 1612 may analyze thecontrols to determine whether implementing the controls would cause acollision or otherwise not result in a safe or permitted outcome. Insuch an example, when it is determined that the controls may not besafe, or may result in a collision, the controls may be aborted ordiscarded, and the autonomous vehicle 1502 may implement a safetyprocedure to get the autonomous vehicle 1502 to a safe operatingcondition. The safety procedure may include coming to a complete stop,pulling to the side of the road, slowing down until a collision is nolonger likely or imminent, and/or another safety procedure. In examples,when controls from the remote control system 1506 are determined to beunsafe, control by the remote control system 1506 may be transferred, atleast temporarily, back to the autonomous vehicle 1502.

In some examples, such as the example in FIG. 16 , the obstacleavoidance component(s) 1612 may be located after the controlcomponent(s) 1610 in the drive stack 1508 (e.g., in order to receivedesired controls from the control component(s) 1610, and test thecontrols for obstacle avoidance). However, even though the obstacleavoidance component(s) 1612 are shown stacked on top of (e.g., withrespect to an autonomous driving software stack) the planningcomponent(s) 1608 and the control component(s) 1610, this is notintended to be limiting. For example, the obstacle avoidancecomponent(s) 1612 may be additionally or alternatively implemented priorto either of the planning component(s) 1608 or the control component(s)1610, prior to the control component(s) 1610 but after the planningcomponent(s) 1608, as part of or integral to the planning component(s)1608 and/or the control component(s) 1610, as part of one or more of theperception component(s) 1604, and/or at a different part of the drivestack 1508 depending on the embodiment. As such, the obstacle avoidancecomponent(s) 1612 may be implemented in one or more locations within anautonomous vehicle driving stack or architecture without departing fromthe scope of the present disclosure.

In some examples, as described herein, the obstacle avoidancecomponent(s) 1612 may be implemented as a separate, discrete feature ofthe autonomous vehicle 1502. For example, the obstacle avoidancecomponent(s) 1612 may operate separately (e.g., in parallel with, priorto, and/or after) the planning layer, the control layer, the actuationlayer, and/or other layers of the drive stack 1508.

The encoder 1616 may encode the sensor data from the sensor manager 1602and/or the sensor(s) 1510 of the autonomous vehicle 1502. For example,the encoder 1616 may be used to convert the sensor data from a firstformat to a second format, such as a compressed, down sampled, and/orlower data size format that the first format. In such an example, thefirst format may be a raw format, a lossless format, and/or anotherformat that includes more data (e.g., for image data, the first formatmay include a raw image format, that may include enough data to fullyrepresent each frame of video). The second format may be in a formatthat includes less data, such as a lossy format and/or a compressedformat (e.g., for image data, the second format may be H264, H265,MPEG-4, MP4, Advanced Video Coding High Definition (AVCHD), Audio VideoInterleave (AVI), Windows Media Video (WMV), etc.). The sensor data maybe compressed to a smaller data size in order to ensure efficient andeffective transmission of the sensor data over the network(s) 1504(e.g., cellular networks, such as 5G).

Once the sensor data is encoded by the encoder 1616, a communicationcomponent 1618 of the autonomous vehicle 1502 may transmit or send theencoded sensor data to the remote control system 1506. Although thesensor data is described as being transmitted as encoded sensor data,this is not intended to be limiting. In some examples, there may not bean encoder 1616, and/or at least some of the sensor data may betransmitted in an uncompressed or non-encoded format.

The remote control system 1506 may receive the sensor data atcommunication component 1624 of the remote control system 1506. Where acommunication is received and/or transmitted as a network communication,the communication component 1618 and/or communication component 1624 maycomprise a network interface which may use one or more wirelessantenna(s) and/or modem(s) to communicate over one or more networks. Byincluding one or more modems and/or one or more wireless antennas, theautonomous vehicle 1502 may be capable of communication across differentnetwork types (e.g., Wi-Fi, cellular 4G, LTE, 5G, etc.), and may alsohave redundancy for when one or more networks may not be available, whenone or more networks may not have a strong enough connection to transmitthe sensor data, and/or for when one or more of the modems goes offlineor stops working. For example, the network interface may be capable ofcommunication over Long-Term Evolution (LTE), Wideband Code-DivisionMultiple Access (WCDMA), Universal Mobile Telecommunications Service(UMTS), Global System for Mobile communications (GSM), CDMA2000, etc.The network interface may also enable communication between objects inthe environment (e.g., vehicles, mobile devices, etc.), using local areanetwork(s), such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave,ZigBee, etc., and/or Low Power Wide-Area Network(s) (LPWANs), such asLong Range Wide-Area Network (LoRaWAN), SigFox, etc.

In some examples, such as where the network strength is below athreshold, or a certain network type is not available for connection(e.g., only a 4G cellular connection is available, and 5G ispreferable), only required or necessary sensor data may be transmittedto the remote control system 1506 (or required or necessary sensor datamay be prioritized in fitting the sensor data into network constraints).For example, during standard or normal operation, all of the sensor datamay be transmitted to the remote control system 1506 (e.g., sensor datafrom each of the sensor(s) 1510 that generate sensor data for use by theremote control system 1506). However, once the network signal dropsbelow a threshold signal strength, or once a certain network typebecomes unavailable, less sensor data, such as sensor data from a subsetof the sensors 110, may be transmitted.

In such examples, orientation data representative of an orientation ofthe VR headset 1516 of the remote control system 1506 may be used. Forexample, if the remote operator is looking toward the left-front of thevirtual vehicle within the virtual environment, the sensor data from thesensor(s) 1510 that have a field(s) of view of the left-front of theautonomous vehicle 1502 may be determined. These sensor(s) 1510 may be aleft-facing camera(s), a forward-facing camera(s), a LIDAR sensor and/orRADAR sensor(s) with a field(s) of view to the left and/or front of theautonomous vehicle 1502 and/or other sensor types. The orientation datamay be used to inform the autonomous vehicle 1502 (e.g., via one or moresignals) of a subset of the sensor data that should be transmitted tothe remote control system 1506. As a result (e.g., based on thesignal(s)), the subset of the sensor data may be encoded and transmittedacross the network(s) 1504 to the remote control system 1506. As theremote operator continues to look around the virtual environment,updated orientation data may be generated and transmitted over thenetwork(s) 1504 to the autonomous vehicle 1502, and updated subsets ofthe sensor data may be received by the remote control system 1506. As aresult, the remote operator may be presented with a field of view thatincludes information relevant to where the remote operator is looking,and the other portions of the virtual environment may not be streamed orrendered.

In some examples, a subset of the sensor data may be transmitted to theremote control system 1506 that enables the virtual environment 1528 tobe rendered without providing any image data (e.g., images or video ofthe real-world or physical environment). For example, locations ofobjects, surfaces, and/or structures, as well as types of objects,surfaces, and/or structures may be determined from the sensor data, andthis information may be transmitted to the remote control system 1506for generating a completely synthetic virtual environment (e.g., noimages or video of the real or physical world, just a virtual world). Insuch an example, if it is determined a vehicle is to the left of theautonomous vehicle 1502, and a person is to the right, the virtualenvironment may be rendered to include a vehicle and a person (e.g.,generic representations) at locations that correspond to the real-world.In a more detailed example, the vehicle type of the vehicle may bedetermined, and the virtual environment may include a virtualrepresentation of the vehicle type (e.g., as determined from a datastore).

In other examples, a combination of a fully rendered virtual environmentand image data (e.g., images or video) may be used within the virtualenvironment. For example, images or video may be included within thevirtual environment in a field of view of the remote operator, but otherportions of the virtual environment may include only virtualrepresentations. As a result, if a remote operator changes orientation,and image data has not yet been received for the updated field of viewof the remote operator, there may still be enough information within theenvironment (e.g., the virtual representations of the objects, surfaces,and/or structures) based on the rendering to allow the remote operatorto control the autonomous vehicle 1502 safely.

Although the signal strength or connection type is described as a reasonfor transmitting only a subset of the sensor data, this is not intendedto be limiting. For example, the subset of the sensor data may betransmitted at all times, regardless of network connection strengthand/or type, in order to reduce bandwidth or preserve network resources.

In some examples, once received by the remote control system 1506, thesensor data (e.g., encoded sensor data) may be decoded by decoder 1626of the remote control system 1506. In other examples, the encoded sensordata may be used by the virtual environment generator 1518 and/or theremote control(s) 1514 (e.g., for calibration) without decoding. Thevirtual environment generator 1518 may use the sensor data to generatethe virtual environment. The sensor data may include image data fromcamera(s), LIDAR data from LIDAR sensor(s), RADAR data from RADARsensor(s), and/or other data types from other sensor(s) 1510, such asvehicle state data and/or configuration data, as described herein. Thevirtual environment generator 1518 may use the sensor data to generateor render the virtual environment and at least a portion of the virtualenvironment may be displayed on a display of the VR headset 1516.Examples of the virtual environment are described in more detail herein,such as with reference to FIG. 17 and FIG. 18 .

In some examples, the virtual environment may be generated using thevehicle state data and/or the calibration data, in addition to imagedata, LIDAR data, SONAR data, etc. In such examples, the vehicle statedata may be used to update a location and/or orientation of the virtualvehicle in the virtual environment and/or to update visual indicators ofthe vehicle state in the virtual environment (e.g., to update aspeedometer, a revolutions per minute (RPM) display, a fuel leveldisplay, a current time where the autonomous vehicle 1502 is located, anodometer, a tachometer, a coolant temperature gauge, a battery chargeindicator, a gearshift indicator, a turn signal indicator, aheadlight/high beam indicator, a malfunction/maintenance indicator,etc.). As a further example, the vehicle state data may be used to applyone or more rendering effects to the virtual environment, such as motionblur that is based at least in part on the velocity and/or accelerationof the autonomous vehicle 1502.

In some examples, state data may be determined by the autonomous vehicle1502 for the objects and surface in the environment, and this stateinformation may be used to generate the virtual environment (e.g., toprovide visual indicators of types of objects, such as persons,vehicles, animals, inanimate objects, etc., or surfaces, such as a pavedroad, a gravel road, an uneven road, an even road, a driveway, a one-waystreet, a two-way street, etc., to provide visual indicators aboutobjects, such as speeds of objects, directions of objects, etc., and/orother information pertaining to the environment).

The calibration data may be used to update the virtual controls (e.g.,the representation of the remote control(s) 1514 in the virtualenvironment). For some non-limiting examples, if the steering wheel isturned to the left, the virtual steering wheel may be rendered as turnedto the left, if the wheels are turned to the right, the virtual wheelsmay be rendered to be turned to the right, if the windows are down, thevirtual windows may be rendered to be down, if the seats are in acertain position, the virtual seats may be rendered to be in the certainpositions, if the instrument panel and/or HMI display is on, at acertain light level, and/or showing certain data, the virtual instrumentpanel and/or HMI display may be on, at the certain light level, and/orshowing the certain data in the virtual environment.

Any other examples for updating the virtual environment to reflect theautonomous vehicle 1502 and/or other aspects of the real-worldenvironment are contemplated within the scope of the present disclosure.By updating at least a portion of the virtual vehicle and/or otherfeatures of the virtual environment using the calibration data, theremote operator may have a more immersive, true-to-life, and realisticvirtual environment to control the virtual vehicle within, therebycontributing to the ability of the remote operator to control theautonomous vehicle 1502 in the real-world environment more safely andeffectively.

At least some of the sensor data may be used by the remote control(s)1514, such as the calibration data for calibrating the remote control(s)1514. For example, similar to described herein with respect to updatingthe virtual environment using the calibration data, the remotecontrol(s) 1514 may be calibrated using the calibration data. In someexamples, a steering component (e.g., a steering wheel, a joystick,etc.) of the remote control(s) 1514 may be calibrated to an initialposition that corresponds to the position of steering component(s) 1628of the autonomous vehicle 1502 at the time of transfer of the control tothe remote control system 1506. In another example, the steeringcomponent sensitivity may be calibrated using the calibration data, suchthat inputs to the steering component of the remote control(s) 1514(e.g., turning the steering wheel×number of degrees to the left)substantially correspond to the inputs to the steering component(s) 1628of the autonomous vehicle 1502 (e.g., the resulting actuation of theautonomous vehicle 1502 may correspond to turning the steering wheel ofthe autonomous vehicle 1502×number of degrees to the left). Similarexamples may be implemented for the acceleration component and/or thebraking component of the remote control(s) to correspond to thesensitivity, degree of movement, pedal stiffness, and/or othercharacteristics of acceleration component(s) 1632 and brakingcomponent(s) 1630, respectively, of the autonomous vehicle 1502. In someexamples, any of these various calibrations may be based at least inpart on the year, make, model, type, and/or other information of theautonomous vehicle 1502 (e.g., if the autonomous vehicle 1502 is a YearN, Make X, Model Y, the virtual vehicle may retrieve associatedcalibration settings from a data store).

In some examples, the calibration data may be used calibrate the remotecontrol(s) 1514 such that the remote control(s) are scaled to theautonomous vehicle 1502 (or object, such as a robot), such as where thevehicle is larger, smaller, or of a different type than the virtualvehicle. For example, the autonomous vehicle 1502 or object may be asmall vehicle or object (e.g., that cannot fit passengers), such as amodel car or an exploratory vehicle (e.g., for navigating into tight orconstrained environments, such as tunnels, beneath structures, etc.),etc., or may be a larger object, such as a bus, a truck, etc. In suchexamples, calibration data may be used to scale the remote control(s)1514 to that of the smaller, larger, or different type of object orvehicle. For example, providing an input to the steering component ofthe remote control(s) 1514, such as by turning a steering wheel 10degrees, may be scaled for a smaller vehicle to 2 degrees, or for alarger vehicle, to 20 degrees. As another example, the braking componentof the remote control(s) 1514 may correspond to anti-skid brakingcontrol inputs, but the autonomous vehicle 1502 or object, especiallywhen small, may use skid braking. In such examples, the remotecontrol(s) may be calibrated such that inputs to the braking componentof the remote control(s) is adjusted for skid braking.

The scaling may additionally, or alternatively, be performed on theoutputs of the remote control(s) (e.g., the control data). For example,after the control inputs to the remote control(s) 1514, the controlinputs may be scaled to correspond to the control(s) of the smaller,larger, or different type of autonomous vehicle 1502 or object. This mayallow the remote operator to control the virtual vehicle or object usingthe remote control(s) 1514 in a way that feels more natural to theremote operator, but while calibrating or scaling the control datarepresentative of the control inputs for the autonomous vehicle 1502 orother object to correspond to the vehicle control data that is useablefor the autonomous vehicle 1502 or other object. In some examples, thismay be performed by the encoder 1620 of the remote control system 1506,and/or by another component.

In any example, prior to transmission of the control data to theautonomous vehicle 1502, the control data may be encoded by the encoder1620. The encoded control data may be in a format that is useable to thevehicle (e.g., the control data from the remote control(s) 1514 may beencoded to generate vehicle control data that is useable by theautonomous vehicle 1502). In other examples, the control data may betransmitted to the autonomous vehicle 1502 over the network(s) 1504using the communication communication component 1624 and communicationcomponent 1618, and the autonomous vehicle 1502 may encode the controldata to generate the vehicle control data. As such, the control datafrom the remote control(s) 1514 may be converted to the vehicle controldata prior to transmission by the remote control system 1506, afterreceipt by the autonomous vehicle 1502, or a combination thereof.

The control data, in some examples, may be received by the communicationcomponent 1618 of the autonomous vehicle 1502 and decoded by the decoder1622. The vehicle control data may then be used by at least one of thelayers of the drive stack 1508 or may bypass the drive stack 1508 (e.g.,where full control is transferred to the remote control system 1506 andthe autonomous vehicle 1502 exits self-driving or autonomous modecompletely) and be passed directly to the control components of theautonomous vehicle 1502, such as the steering component(s) 1628, thebraking component(s) 1630, the acceleration component(s) 1632, and/orother components (e.g., a blinker, light switches, seat actuators,etc.). As such, the amount of control given to the remote control system1506 may include from no control, full control, or partial control. Theamount of control of the autonomous vehicle 1502 may inverselycorrespond to the amount of control given to the remote control system1506. Thus, when the remote control system 1506 has full control, theautonomous vehicle 1502 may not execute any on-board control, and whenthe remote control system 1506 has no control, the autonomous vehicle1502 may execute all on-board control.

In examples where the vehicle control data (e.g., corresponding to thecontrol data generated based on control inputs to the remote control(s)1514) is used by the drive stack 1508, there may be different levels ofuse. In some examples, only the obstacle avoidance component(s) 1612 maybe employed. In such examples, the vehicle control data may be analyzedby the obstacle avoidance component(s) 1612 to determine whetherimplementing the controls corresponding to the vehicle control datawould result in a collision or an otherwise unsafe or undesirableoutcome. When a collision or unsafe outcome is determined, theautonomous vehicle 1502 may implement other controls (e.g., controlsthat may be similar to the controls corresponding to the vehicle controldata but that decrease, reduce, or remove altogether the risk ofcollision or other unsafe outcome). In the alternative, the autonomousvehicle 1502 may implement a safety procedure when a collision or otherunsafe outcome is determined, such as by coming to a complete stop. Inthese examples, the control inputs from the remote control(s) 1514 maybe associated (e.g., one-to-one) with the controls of the autonomousvehicle 1502 (e.g., the control inputs to the remote control(s) 1514 maynot be suggestions for control of the vehicle, such as waypoints, butrather may correspond to controls that should be executed by theautonomous vehicle 1502).

As described herein, the control inputs from the remote control(s) 1514may not be direct or one-to-one controls for the autonomous vehicle1502, in some examples. For example, the control inputs to the remotecontrol(s) 1514 may be suggestions. One form of suggestion may be anactual input to a steering component, an acceleration component, abraking component, or another component of the remote control(s) 1514.In such an example, the vehicle control data corresponding to thesecontrol inputs to the remote control(s) 1514 may be used by the drivestack 1508 to determine how much, or to what degree, to implement thecontrols. For example, if the remote operator provides an input to asteering component of the remote control(s) 1514 (e.g., to turn asteering wheel 10 degrees), the planning component(s) 1608 and/or thecontrol component(s) 1610 of the drive stack 1508 may receive thevehicle control data representative of the input to the steeringcomponent, and determine to what degree to turn to the left (or to notturn left at all). The drive stack 1508 may make a determination to turnleft, for example, but may determine that a more gradual turn is safer,follows the road shape or lane markings more accurately, and/orotherwise is preferable over the rate of the turn provided by the remoteoperator (e.g., the 10 degree turn of the steering wheel). As such, thevehicle control data may be updated and/or new vehicle control data maybe generated by the drive stack 1508, and executed by the steeringcomponent(s) 1628 of the autonomous vehicle 1502 (e.g., based at leastin part on a command or signal from the actuation component(s) 1614).

Similar use of the vehicle control data may be performed based at leastin part on inputs to the acceleration component, braking component,and/or other components of the remote control(s) 1514. For example, aninput to an acceleration component of the remote control(s) 1514 maycause an acceleration by the acceleration component(s) 1632 of theautonomous vehicle 1502, but the acceleration rate may be less, more, orzero, depending on the determination(s) by the drive stack 1508. Asanother example, an input to a braking component of the remotecontrol(s) 1514 may cause a braking by the braking component(s) 1630 ofthe autonomous vehicle 1502, but the deceleration rate may be less,more, or zero, depending on the determination(s) by the drive stack1508.

Another form of suggestions from the remote control(s) 1514 may bewaypoint suggestions. For example, the remote operator may use a remotecontrol(s) 1514 that is a pointer (e.g., a virtual laser pointer), andmay point to virtual locations in the virtual environment that thevirtual vehicle is to navigate to (e.g., a virtual waypoint). Thereal-world locations in the real-world environment that correspond tothe virtual locations in the virtual environment may be determined, andthe vehicle control data may represent the real-world locations (e.g.,the real-world waypoints). As such, the drive stack 1508, such as theplanning component(s) 1608 and/or the control component(s) 1610, may usethe real-world waypoint to determine a path and/or control(s) forfollowing the path to reach the real-world waypoint. The actuationcomponent(s) 1614 may then cause the steering component(s) 1628, thebraking component(s) 1630, the acceleration component(s) 1632, and/orother components of the autonomous vehicle 1502 to control theautonomous vehicle 1502 to travel to the real-world locationcorresponding to the real-world waypoint. The remote operator maycontinue to provide these control inputs to navigate the autonomousvehicle 1502 through the situation, scenario, and/or environment thatnecessitated the transfer of at least partial control to the remotecontrol system 1506.

Now referring to FIG. 17 and FIG. 18 , FIG. 17 and FIG. 18 illustratenon-limiting examples of a virtual environment 1700 that may begenerated by the virtual environment generator 1518. In one or moreembodiments, the virtual environments may be displayed on a display ofthe VR headset 1516. Alternatively, the virtual environments may bedisplayed on a display corresponding to a physical representation of avehicle. The physical representation may include any configuration ofcontrol (e.g., a steering wheel, one or more accelerators or brakes, oneor more transmission controls), seating, or visibility (e.g., one ormore displays positioned as mirrors) features corresponding to physical,real-world counterparts in an ego-vehicle. Virtual environment 1700 ofFIG. 17 may include a virtual environment where an exterior of a vehicle1702 is rendered, such that a field of view of the remote operatorincludes the exterior of the vehicle 1702. In one or more embodiments,the vehicle 1702 may be presented as a virtually simulated vehicle.Alternatively, the virtual environment 1700 may be rendered in one ormore displays positioned around a partially or completely physicalvehicle 1702 calibrated to correspond to the ego-vehicle. In the caseswhere the vehicle 1702 comprises a virtual vehicle, the virtual vehicle1702 may be rendered on a surface 1704 of the virtual environment 1700.In this case, the surface 1704 may be one of any number of suitablesurfaces, such as a representation of a garage floor, a laboratoryfloor, etc. However, this is not intended to be limiting, and in someexamples, the surface 1704 may be rendered to represent the surface theautonomous vehicle 1502 is on in the real-world environment (e.g., usingsensor data generated from cameras with a field(s) of view of thesurface around the autonomous vehicle 1502, such as a parkingcamera(s)).

The sensor data, such as image data, representative of a field(s) ofview of the sensor(s) 1510 may be displayed within the virtualenvironment 1700 on one or more virtual displays 1706, and/or additionalor alternative virtual displays 1706. In some examples, the virtualdisplays 1706 may be rendered to represent up to a 360 degree field ofview of the sensor(s) 1510 of the autonomous vehicle 1502. As describedherein, the surface 1704 and/or an upper portion 1708 of the virtualenvironment 1700 may also be rendered to represent the real-worldenvironment of the autonomous vehicle 1502. The upper portion 1708 mayinclude buildings, trees, the sky, and/or other features of thereal-world environment, such that the virtual environment 1700 mayrepresent a fully immersive environment. The surface 1704 and/or theupper portion 1708, similar to the virtual displays 1706, may includeimages or video from image data generated by the autonomous vehicle1502, may include rendered representations of the environment as gleanedfrom the sensor data (e.g., image data, LIDAR data, RADAR data, etc.),or a combination thereof.

The instance of the virtual environment 1700 illustrated in FIG. 17 mayrepresent the scenario represented in the image 1520. For example, thevirtual display 206B may include the virtual representations of the van1524, the unloaded boxes 1526, the divided street 1548, and/or otherfeatures of the image 1520. The virtual representations of the imagedata may include the images or video from the image data, renderedwithin the virtual environment 1700. As such, the images or videodisplayed on the virtual displays 1706 may be the actual images or video(e.g., not a virtual representation thereof). In other examples, theimages or video displayed on the virtual displays 1706 may be a renderedrepresentation of the environment, which may be generated from thesensor data (e.g., the image data, the LIDAR data, the SONAR data,etc.).

As described herein, the vehicle state data and/or the calibration datamay be used to generate the virtual environment. In such examples,wheels 1710 of the virtual vehicle 1702 may be rendered at approximatelythe wheel angle of the wheels of the autonomous vehicle 1502 in thereal-world environment. In this illustration, the wheels may bestraight. Similarly, lights may be turned on or off, including brakelights when braking, emergency lights when turned on, etc. When thevehicle 1702 includes a physical, tangible representation, the vehiclestate data and/or the calibration data of the ego-vehicle may be used tocalibrate and orient the physical representation vehicle 1702.

When controlling a virtual vehicle 1702 implemented as a virtual vehiclein the virtual environment 1700, or other virtual environments where thevantage point of the remote operator is outside of the virtual vehicle1702, the remote operator may be able to move around the virtualenvironment 1700 freely to control the virtual vehicle 1702 fromdifferent vantage points (or may be able to change the vantage point toinside the virtual vehicle, as illustrated in FIG. 18 ). For example,the remote operator may be able to sit on top of or above the virtualvehicle 1702, to the side of the virtual vehicle 1702, in front of thevirtual vehicle 1702, behind the virtual vehicle 1702, etc.

In examples where the remote operator provides virtual waypoints ratherthan actual controls, a vantage point outside of the virtual vehicle1702 may be more useful. For example, the remote operator may have avantage point from on top of the virtual vehicle 1702, such as atlocation 1712 within the virtual environment 1700, and may use device1714 (e.g., a virtual pointer, a virtual laser, etc.) to identify alocation within the virtual environment 1700 and/or a location withinthe image data represented within the virtual environment 1700, such aslocation 1716. When the location 1716 corresponds to the image data,such as a point(s) or pixel(s) within the image data, the real-worldcoordinates corresponding to the point(s) or the pixel(s) may bedetermined (e.g., by the autonomous vehicle 1502 and/or the remotecontrol system 1506). For example, the camera(s) that captured the imagedata may be calibrated such that transformations from two-dimensionallocations of the point(s) or the pixel(s) within the image data tothree-dimensional points in the real-world environment may be computedor known. As a result, the virtual way-points (e.g., the location 1716)identified within the virtual environment 1700 by the remote operatormay be used to determine real-world locations (e.g., corresponding tothe location 1716) for the autonomous vehicle 1502 to navigate to. Asdescribed herein, the autonomous vehicle 1502 may use this informationto determine the path, controls, and/or actuations that will control theautonomous vehicle 1502 to the real-world location.

As the autonomous vehicle 1502 is controlled through the real-worldenvironment, the virtual displays 1706 may be updated to reflect theupdated sensor data over time (e.g., at the frame rate that the sensordata is captured, such as 30 frames per second (“fps”), 60 fps, etc.).As the (virtual) vehicle 1702 is being controlled, the wheels, lights,windows, blinkers, etc., may be updated according to the correspondingfeatures on the autonomous vehicle 1502 in the real-world environment.

Now referring to FIG. 18 , the virtual environment 1528 may be the samevirtual environment 1528 of FIG. 15 described herein. Although thevantage point illustrated in FIG. 18 is from a left-side driver's seatwithin the virtual vehicle, this is not intended to be limiting. Forexample, and without departing from the scope of the present disclosure,the remote operator may have a vantage point from the position aright-side driver's seat (e.g., for jurisdictions where driving is onthe left side of the road), a passenger's seat, a back seat, animaginary seat (e.g., a middle-driver's seat), or from a vantage pointwithin the virtual vehicle not corresponding to a seat, such as fromanywhere within the virtual vehicle.

As described herein, one or more of the features of the virtual vehiclemay be made at least partially transparent and/or may be removed fromthe rendering of the virtual vehicle. For example, certain portions of areal-world vehicle (alternatively referred to herein as “ego-vehicle” or“physical vehicle”) may be used for structural support, but may causeocclusions for a driver (e.g., “blind spots). In a virtual vehicle, thisneed for structural support is non-existent, so portions of the virtualvehicle that may be visually occluding may be removed and/or made atleast partially transparent in the virtual environment 1528. Forexample, the support column 1538, and/or other support columns of thevirtual vehicle, may be made transparent (as illustrated in FIG. 18 ) ormay be removed completely from the rendering. In other examples, doors1804 may be made transparent (e.g. but for an outline) or entirelyremoved. As a result, the remote operator may be presented with afield(s) of view that is more immersive, with less occlusions, therebyfacilitating more informed, safer control.

In addition, a portion(s) of the virtual vehicle may be made at leastpartially transparent or be removed even where the portion(s) of thevirtual vehicle does not cause occlusions, in order to allow the remoteoperator to visualize information about the virtual vehicle (and thusthe autonomous vehicle 1502) that would not be possible in a real-worldenvironment. For example, a portion of the virtual vehicle between avantage point of the remote operator and one or more of the wheelsand/or tires of the vehicle may be made at least partially transparentor may be removed from the rendering, such that the remote operator isable to visualize an angle of the wheel(s) and/or the tire(s) (e.g.,where the wheels and/or tire s are at the angle based on the calibrationdata).

The virtual environment 1528 may include, in addition to oralternatively from the features described herein with respect to FIG. 15, a virtual instrument panel 1806, virtual side-view or wing-mirrors1802, and/or other features. The virtual instrument panel 1806 maydisplay any number of different information, such as, withoutlimitation, a speedometer, a fuel level indicator, an oil pressureindicator, a tachometer, an odometer, turn indicators, gearshiftposition indicators, seat belt warning light(s), parking-brake warninglight(s), engine-malfunction light(s), airbag (SRS) system information,lighting controls, safety system controls, navigation information, etc.The virtual side-view or wing-mirrors 1802 may display sensor datacaptured by one or more sensor(s) 1510 (e.g., camera(s)) of theautonomous vehicle 1502 with a field(s) of view to the rear and/or tothe side of the autonomous vehicle 1502 (e.g., to represent a side-viewor wing-mirror of the autonomous vehicle 1502).

Now referring to FIG. 19 and FIG. 20 , each block of method 1900 andmethod 2000, described herein, comprises a computing process that may beperformed using any combination of hardware, firmware, and/or software.For instance, various functions may be carried out by a processorexecuting instructions stored in memory. The methods may also beembodied as computer-usable instructions stored on computer storagemedia. The methods may be provided by a standalone application, aservice or hosted service (standalone or in combination with anotherhosted service), or a plug-in to another product, to name a few. Inaddition, method 1900 and method 2000 are described, by way of example,with respect to the autonomous vehicle control system 1500 of FIG. 15and FIG. 16 . However, these methods may additionally or alternativelybe executed by any one system, or any combination of systems, including,but not limited to, those described herein.

FIG. 19 is a flow diagram showing a method 1900 of remote control of anautonomous vehicle, in accordance with some embodiments of the presentdisclosure. The method 1900, at block 1902, includes determining totransfer at least partial control of a vehicle to a remote controlsystem. For example, the autonomous vehicle 1502 (e.g., one or morecomponents of the drive stack 1508), the remote operator, a passenger,and/or another actor may determine to transfer at least partial controlto the remote control system 1506. In such examples, the determinationmay be to activate, initiate, or otherwise begin a remote controlsession. In examples where the autonomous vehicle 1502 determines totransfer control, the determination, as described herein, may be basedon a constraint on the autonomous vehicle 1502 such as rules of theroad, an obstacle in a path of the autonomous vehicle 1502, etc., thatmay not allow the autonomous vehicle 1502 to navigate a situation,scenario, and/or environment. The determination may be made based on ananalysis of sensor data of the autonomous vehicle 1502, and may be madeby one or more layers of the drive stack 1508 in some examples.

The method 1900, at block 1904, includes receiving sensor data from asensor(s) of the vehicle. For example, sensor data from the sensor(s)1510 may be received.

The method 1900, at block 1906, includes encoding the sensor data togenerate encoded sensor data. For example, the sensor data may beencoded into a different format, such as a less data intense format. Ifthe sensor data includes image data, for example, the image data may beconverted from a first format (e.g., a raw image format) to a secondformat (e.g., an encoded video format, such as H.264, H.265, AV1, VP9,or another image format, including but not limited to those describedherein).

The method 1900, at block 1908, includes transmitting the encoded sensordata to the remote control system for display by a virtual realityheadset of the remote control system. For example, the encoded sensordata may be transmitted to the remote control system 1506 for display ona display of the VR headset 1516.

The method 1900, at block 1910, includes receiving control datarepresentative of at least one control input to the remote controlsystem. For example, control data representative of at least one inputto the remote control(s) 1514 may be received from the remote controlsystem 1506. In some examples, the control data may not be in a formatuseable by the autonomous vehicle 1502, and thus may be converted orencoded to vehicle control data useable by the autonomous vehicle 1502.In other examples, the control data may be useable by the autonomousvehicle 1502, or may have already been encoded by the remote controlsystem 1506 and thus the control data received may include the vehiclecontrol data.

The method 1900, at block 1912, includes causing actuation of anactuation component(s) of the vehicle. For example, the control data(and/or the vehicle control data) may be used by the autonomous vehicle1502 to cause actuation of at least one actuation component of theautonomous vehicle 1502, such as the steering component(s) 1628, thebraking component(s) 1630, and/or the acceleration component(s) 1632.

FIG. 20 is an example flow diagram for a method 2000 of remote controlof an autonomous vehicle, in accordance with some embodiments of thepresent disclosure. The method 2000, at block 2002, includes receivingsensor data representative of a field of a view in a physicalenvironment of a sensor(s) of a vehicle. For example, sensor datarepresentative of a field(s) of view of the sensor(s) 1510 of theautonomous vehicle 1502 in the real-world environment may be received.

The method 2000, at block 2004, includes receiving vehicle stateinformation of the vehicle. For example, the vehicle state informationmay be received from the autonomous vehicle 1502.

The method 2000, at block 2006, includes generating a virtualenvironment. For example, the virtual environment generator 1518 maygenerate a virtual environment based on the sensor data, the vehiclestate data, and/or calibration data.

The method 2000, at block 2008, includes causing display of the virtualenvironment on a display of a remote control system. For example, thevirtual environment may be displayed on a display of the VR headset 1516of the remote control system 1506.

The method 2000, at block 2010, includes generating control datarepresentative of a virtual control(s) of the vehicle. For example,control data representative of control input(s) to the remote control(s)1514 for controlling a virtual vehicle may be generated.

The method 2000, at block 2012, includes transmitting the control datato the vehicle. For example, the control data may be transmitted to theautonomous vehicle 1502. In some examples, prior to transmission, thecontrol data may be encoded to create vehicle control data useable bythe autonomous vehicle 1502.

Now referring to FIG. 21 , each block of method 2100, described herein,comprises a computing process that may be performed using anycombination of hardware, firmware, and/or software. For instance,various functions may be carried out by a processor executinginstructions stored in memory. The methods may also be embodied ascomputer-usable instructions stored on computer storage media. Themethods may be provided by a standalone application, a service or hostedservice (standalone or in combination with another hosted service), or aplug-in to another product, to name a few. In addition, method 2100 isdescribed, by way of example, with respect to the autonomous vehiclecontrol system 1500 of FIG. 15 and FIG. 16 . However, these methods mayadditionally or alternatively be executed by any one system, or anycombination of systems, including, but not limited to, those describedherein.

FIG. 21 is an example signal flow diagram for a method 2100 of remotecontrol of an autonomous vehicle, in accordance with some embodiments ofthe present disclosure. The method 2100, as illustrated in FIG. 21 , maybegin at the top of the page and end at the bottom. However, this is notintended to be limiting, and one or more of the blocks may be inalternative order and/or may be removed, or one or more additional oralternative blocks may be used in the method 2100 without departing fromthe scope of the present disclosure.

The method 2100, at block 2102, includes transferring at least partialcontrol. For example, at least partial control may be transferred by theautonomous vehicle 1502 to the remote control system 1506. In suchexamples, a signal(s), S1, may be generated and transmitted from theautonomous vehicle 1502 (e.g., via the communication component 1624) tothe remote control system 1506 (e.g., via the communication component1618) to inform the remote control system 1506 that at least partialcontrol is being transferred. The signal(s), S1, may be representativeof data indicating that control is being transferred. The transfer ofcontrol may not be executed, in some examples, as illustrated by thedashed lines. For example, where control of the vehicle 102 or object isalways performed by the remote control system 1506, there may not be aneed to transfer control.

The method 2100, at block 2104, includes generating and transmittingcalibration data. For example, the sensor(s) 1510 of the autonomousvehicle 1502 may generate calibration data, and the autonomous vehicle1502 may transmit the calibration data to the remote control system 1506(e.g., via the communication component 1624 or communication component1618). In such examples, a signal(s), S2, may be generated andtransmitted from the autonomous vehicle 1502 to the remote controlsystem 1506 that represents the calibration data. The generating andtransmitting of the calibration data may not be executed, in someexamples, as illustrated by the dashed lines.

The method 2100, at block 2112, includes calibrating remote control(s).For example, the calibration data received via the signal(s), S2, may beused by the remote control system 1506 to calibrate the remotecontrol(s) 1514. The calibrating of the remote control(s) 1514 may notbe executed, in some examples, as illustrated by the dashed lines.

The method 2100, at block 2106, includes generating and transmittingsensor data and/or vehicle state data. For example, the sensor(s) 1510of the autonomous vehicle 1502 may generate sensor data and/or vehiclestate data and the autonomous vehicle 1502 may transmit the sensor dataand/or the vehicle state data to the remote control system 1506 (e.g.,via the communication component 1624 and/or communication component1618). In such examples, a signal(s), S3, may be generated andtransmitted from the autonomous vehicle 1502 to the remote controlsystem 1506 that represents the sensor data and/or the vehicle statedata.

The method 2100, at block 2114, includes rendering a virtualenvironment. For example, the virtual environment generator 1518 maygenerate and/or render the virtual environment based on the sensor data,the vehicle state data, and/or the calibration data.

The method 2100, at block 2116, includes displaying the virtualenvironment on a VR headset. For example, the virtual environment, or atleast a portion thereof, may be displayed on the VR headset 1516 of theremote control system 1506.

The method 2100, at block 2118, includes receiving control input(s) toremote control(s). For example, the remote operator may provide one ormore control inputs to the remote control(s) 1514.

The method 2100, at block 2120, includes generating and transmittingcontrol data. For example, the remote control(s) 1514 of the remotecontrol system 1506 may generate control data based on the controlinput(s) and the remote control system 1506 may transmit the controldata to the autonomous vehicle 1502 (e.g., via the communicationcomponent 1624 and/or communication component 1618). In such examples, asignal(s), S4, may be generated and transmitted from the remote controlsystem 1506 to the autonomous vehicle 1502 that represents the controldata.

The method 2100, at block 2108, includes determining vehicle controldata based on the control data. For example, the autonomous vehicle 1502may determine whether the control data is useable by the autonomousvehicle 1502 and, if not, may generate vehicle control data thatcorresponds to the control data but that is useable by the autonomousvehicle 1502.

The method 2100, at block 2110, includes executing control(s) based onthe vehicle control data. For example, one or more controls may beexecuted by the autonomous vehicle 1502 that may correspond to thecontrol input(s) to the remote control system 1506.

Now referring to FIG. 22 , FIG. 22 is an example data flow diagramillustrating a process 2200 for training an autonomous vehicle using amachine learning model(s), in accordance with some embodiments of thepresent disclosure. Any number of inputs, including but not limited tosensor data 2202 and/or control data representative of control input(s)to remote control(s) 1514 of the remote control system 1506, may beinput into a machine learning model(s) 2204.

The machine learning model(s) 2204 may generate or compute any number ofoutputs, including but not limited to vehicle control datarepresentative of vehicle control(s) 2206 for controlling the autonomousvehicle 1502. In some examples, the output may be control data, such asthe control data generated by the remote control(s) 1514 of the remotecontrol system 1506, and the control data may be, where necessary,encoded or otherwise converted to vehicle control data representative ofthe vehicle control(s) 2206 useable by the autonomous vehicle 1502. Insome examples, the vehicle control(s) 2206 may include vehicletrajectory information, such as a path, or points along a path, that theautonomous vehicle 1502 should navigate along within the environment.The vehicle control(s) 2206 may be transmitted or sent to a controlcomponent(s) 1610, planning component(s) 1608, and/or other layers ofthe drive stack 1508, and the control component(s) 1610, the planningcomponent(s) 1608, and/or other layers of the drive stack 1508 may usethe vehicle control(s) 1512 to control the autonomous vehicle 1502according to the vehicle control(s) 2206.

The sensor data 2202 may be image data, LIDAR data, SONAR data, and/ordata from one or more other sensor(s) 1510 of the autonomous vehicle1502 that may be representative of the real-world environment of theautonomous vehicle 1502. In some examples, the sensor data may furtherinclude vehicle state data representative of the state of the vehicle102, such as speed, velocity, acceleration, deceleration, orientation orpose, location or position in the environment and/or other statusinformation. This data may be captured by and/or received from one ormore of the sensors 110 of the autonomous vehicle 1502, such as one ormore of the IMU sensor(s) 2508, speed sensor(s) 2536, steering sensor(s)2524, vibration sensor(s) 2534, and/or one or more sensors of the brakesensor system 2532, propulsion system 2526, and/or steering system 2516.The vehicle state data (e.g., speed, orientation, etc.) may be valuableto the machine learning model(s) 2204 in computing the vehiclecontrol(s) 2206 as the vehicle state data may inform the machinelearning model(s) 2204 as to what vehicle control(s) 2206 are mostuseful given the current vehicle state.

For example, the autonomous vehicle 1502 may transfer at least partialcontrol to the remote control system 1506 as a result of encountering asituation, scenario, and/or environment that the autonomous vehicle 1502is not permitted to handle autonomously (e.g., due to one or moreconstraints). A remote operator may control the virtual vehicle throughthe virtual environment, and the control inputs by the remote operatormay be represented by control data. The control data may then be encodedor converted to vehicle control data useable by the autonomous vehicle1502, and the autonomous vehicle 1502 may be controlled through thesituation, scenario, and/or environment based on the vehicle controldata. Throughout the remote control session, the sensor(s) 1510 of theautonomous vehicle 1502 may generate sensor data 2202. The sensor data2202 (e.g., image data and/or vehicle state data) may be input into themachine learning model(s) 2204, and the machine learning model(s) 2204may learn (e.g. using ground truth control data) the vehicle control(s)2206 for navigating the situation, scenario, and/or environment, and/orsimilar situations, scenarios, and/or environments, such that during anext occurrence, the autonomous vehicle 1502 may be able to navigateitself through the situation, scenario, and/or environment without theneed for the remote control system 1506.

The machine learning model(s) 2204 may include any type of machinelearning model(s), such as machine learning models using linearregression, logistic regression, decision trees, support vector machines(SVM), Naïve Bayes, k-nearest neighbor (Knn), K means clustering, randomforest, dimensionality reduction algorithms, gradient boostingalgorithms, neural networks (e.g., auto-encoders, convolutional,recurrent, perceptrons, long/short terms memory, Hopfield, Boltzmann,deep belief, deconvolutional, generative adversarial, liquid statemachine, etc.), and/or other types of machine learning models.

Now referring to FIG. 23 , FIG. 23 is an example illustration of amachine learning model(s) for training an autonomous vehicle accordingto the process of FIG. 22 , in accordance with some embodiments of thepresent disclosure. The machine learning model(s) 2204 of FIG. 23 may beone example of a machine learning model(s) that may be used in theprocess 2200. However, the machine learning model(s) 2204 of FIG. 23 isnot intended to be limiting, and the machine learning model(s) 2204 mayinclude additional and/or different machine learning models than themachine learning model(s) 2204 of FIG. 23 . The machine learningmodel(s) 2204 of FIG. 23 may include a convolutional neural network andthus may alternatively be referred to herein as convolutional neuralnetwork or convolutional network.

The convolutional network includes the sensor data 2202 representativeof one or more images generated based on image data from one or morecamera(s) of the autonomous vehicle 1502. In some examples, theconvolutional network may also include other inputs as sensor data, suchas LIDAR data, RADAR data, vehicle state data, etc. The sensor data 2202may be input into convolutional stream(s) 2302 of the convolutionalnetwork. For example, sensor data from each sensor (e.g., where two ormore sensors are used) may be input its own convolutional stream(s)2302.

A convolutional stream(s) 2302 may include any number of layers, such asthe layers 2306. One or more of the layers may include an input layer.The input layer may hold values associated with the sensor data. Forexample, the input layer may hold values representative of the raw pixelvalues of the image(s) input to the convolutional network as a volume(e.g., a width, a height, and color channels (e.g., RGB), such as32×32×3).

One or more layers may include convolutional layers. The convolutionallayers may compute the output of neurons that are connected to localregions in an input layer (e.g., the input layer), each computing a dotproduct between their weights and a small region they are connected toin the input volume. A result of the convolutional layers may be anothervolume, with one of the dimensions based at least in part on the numberof filters applied (e.g., the width, the height, and the number offilters, such as 32×32×12, if 12 were the number of filters).

One or more of the layers may include a rectified linear unit (ReLU)layer. The ReLU layer(s) may apply an elementwise activation function,such as the max (0, x), thresholding at zero, for example. The resultingvolume of a ReLU layer may be the same as the volume of the input of theReLU layer.

One or more of the layers may include a pooling layer. The pooling layermay perform a down sampling operation along the spatial dimensions(e.g., the height and the width), which may result in a smaller volumethan the input of the pooling layer (e.g., 16×16×12 from the 32×32×12input volume).

One or more of the layers may include a fully connected layer. Eachneuron in the fully connected layer(s) may be connected to each of theneurons in the previous volume. The fully connected layer may computeclass scores, and the resulting volume may be 1×1×number of classes. Insome examples, the convolutional stream(s) 2302 may include a fullyconnected layer, while in other examples, a fully connected fullyconnected layer(s) 2304 of the convolutional network 504 may be thefully connected layer for the convolutional stream(s) 2302.

Although input layers, convolutional layers, pooling layers, ReLUlayers, and fully connected layers are discussed herein with respect tothe convolutional stream(s) 2302, this is not intended to be limiting.For example, additional or alternative layers may be used in theconvolutional stream(s) 2302, such as normalization layers, SoftMaxlayers, and/or other layer types. Further, the order and number oflayers of the convolutional network and/or the convolutional stream(s)2302 is not limited to any one architecture.

In addition, some of the layers may include parameters (e.g., weights),such as the convolutional layers and the fully connected layers, whileothers may not, such as the ReLU layers and pooling layers. In someexamples, the parameters may be learned by the convolutional stream(s)2302 and/or the fully connected layer(s) 2304 during training. Further,some of the layers may include additional hyper-parameters (e.g.,learning rate, stride, epochs, etc.), such as the convolutional layers,the fully connected layers, and the pooling layers, while other layersmay not, such as the ReLU layers. The parameters and hyper-parametersare not to be limited, and may differ depending on the embodiment.

The output of the convolutional stream(s) 2302 may be input to the fullyconnected layer(s) 2304 of the convolutional network. In addition to theoutput of the convolutional stream(s) 2302, variable(s), at least someof which may be representative of the vehicle state, may be input to thefully connected layer(s) 2304.

The machine learning model(s) 2204 may be trained using example controldata (e.g., vehicle control data, trajectories, etc.) as ground truthdata and/or sensor data 2202 for given inputs to the machine learningmodel(s) 2204. In some examples, the control data may be based on thecontrol inputs to the remote control(s) 1514 of the remote controlsystem 1506, and/or based on the vehicle control data generated as aresult of the control inputs. In some examples, the training data maycorrespond to a virtual vehicle, such as a vehicle driven in a virtualsimulation comprising a virtual environment.

Now referring to FIG. 24 , each block of method 2400, described herein,comprises a computing process that may be performed using anycombination of hardware, firmware, and/or software. For instance,various functions may be carried out by a processor executinginstructions stored in memory. The methods may also be embodied ascomputer-usable instructions stored on computer storage media. Themethods may be provided by a standalone application, a service or hostedservice (standalone or in combination with another hosted service), or aplug-in to another product, to name a few. In addition, method 2400 maybe executed by any one system, or any combination of systems, includingbut not limited to those described herein.

FIG. 24 is an example flow diagram for a method 2400 of training anautonomous vehicle using a machine learning model(s), in accordance withsome embodiments of the present disclosure. The method 2400, at block2402, includes receiving control data supplied from a remote controlsystem representative of control inputs. For example, control datarepresentative of control inputs supplied by a remote control system1506 may be received (e.g., by the remote control system 1506, by amodel training server(s), by the autonomous vehicle 1502, etc.).

The method 2400, at block 2404, includes converting the control data tovehicle control data usable by a vehicle. For example, the control datamay be converted to vehicle control data that is useable by a vehicle(e.g., by the autonomous vehicle 1502).

The method 2400, at block 2406 includes receiving sensor data generatedby one or more sensors of the vehicle during executing of vehiclecontrols corresponding to the vehicle control data by the vehicle. Forexample, sensor data may be received (e.g., by the remote control system1506, the model training server(s), the autonomous vehicle 1502, etc.),where the sensor data is or was generated during execution of vehiclecontrols corresponding to the vehicle control data. The vehicle controlsmay be the controls from the control component(s) 1610, the planningcomponent(s) 1608, the actuation component(s) 1614, the steeringcomponent(s) 1628, the braking component(s) 1630, the accelerationcomponent(s) 1632, and/or other components of the autonomous vehicle1502 (and/or another vehicle or object).

The method 2400, at block 2408, includes applying the vehicle controldata and/or the sensor data to a machine learning model(s). For example,the sensor data (e.g., image data, LIDAR data, SONAR data, vehicle statedata, etc.) may be applied to the machine learning model(s) (e.g., themachine learning model(s) 2204 of FIG. 22 and FIG. 23 ). In someexamples, the sensor data may be applied to the machine learningmodel(s) and the vehicle control data may be used as ground truth datato train the machine learning model(s).

The method 2400, at block 2410, includes computing, by the machinelearning model(s), vehicle control(s). For example, the machine learningmodel(s) may compute vehicle control(s) (e.g., represented as vehiclecontrol data) that correspond to the sensor data.

The method 2400, at block 2412, includes comparing the vehiclecontrol(s) to ground truth data. For example, the ground truth data mayinclude the vehicle control data and/or paths or trajectories throughthe environment as labeled or annotated within the representations ofthe sensor data (e.g., the images).

The method 2400, at block 2414, includes, based on the comparing atblock 2412, updating the machine learning model(s). For example, theparameters (e.g., weights, biases, etc.) of the machine learningmodel(s) may be updated (e.g., using backpropagation, parameter updates,etc.). This process may repeat until the machine learning model(s) hasacceptable or desirable accuracy.

As a result of the method 2400, the machine learning model(s) may betrained such that the machine learning model(s), once deployed, may takesensor data as an input and generate vehicle control(s) for navigatingthrough environments, situations, and/or scenarios without the need forremote control. As such, over time, the remote control system 1506 maybe used to train the autonomous vehicle 1502 (and/or other vehicles orobjects) how to navigate different types of scenarios, situations,and/or environments until the need for remote control, or a remotecontrol system 1506, may become unnecessary.

FIG. 25 is an illustration of an example autonomous vehicle 2500, inaccordance with some embodiments of the present disclosure. Theautonomous vehicle 1502 may include a passenger vehicle, such as a car,a truck, a bus, and/or another type of vehicle that accommodates one ormore passengers. Autonomous vehicles are generally described in terms ofautomation levels, defined by the National Highway Traffic SafetyAdministration (NHTSA), a division of the US Department ofTransportation, and the Society of Automotive Engineers (SAE) “Taxonomyand Definitions for Terms Related to Driving Automation Systems forOn-Road Motor Vehicles” (Standard No. J3016-201806, published on Jun.15, 2018, Standard No. J3016-201609, published on Sep. 30, 2016, andprevious and future versions of this standard). The autonomous vehicle1502 may be capable of functionality in accordance with one or more ofLevel 3-Level 5 of the autonomous driving levels. For example, theautonomous vehicle 1502 may be capable of conditional automation (Level3), high automation (Level 4), and/or full automation (Level 5),depending on the embodiment.

The autonomous vehicle 1502 may include components such as a chassis, avehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, andother components of a vehicle. The autonomous vehicle 1502 may include apropulsion system 2526, such as an internal combustion engine, hybridelectric power plant, an all-electric engine, and/or another propulsionsystem type. The propulsion system 2526 may be connected to a drivetrain of the autonomous vehicle 1502, which may include a transmission,to enable the propulsion of the autonomous vehicle 1502. The propulsionsystem 2526 may be controlled in response to receiving signals from thethrottle/accelerator(s) 2538.

A steering system 2516, which may include a steering wheel, may be usedto steer the autonomous vehicle 1502 (e.g., along a desired path orroute) when the propulsion system 2526 is operating (e.g., when thevehicle is in motion). The steering system 2516 may receive signals fromsteering actuators 2512. The steering wheel may be optional for fullautomation (Level 5) functionality.

The brake sensor system 2532 may be used to operate the vehicle brakesin response to receiving signals from the brake actuators 2510 and/orbrake sensors.

Controller(s) 2544, which may include one or more systems on chips(SoC(s) 2702) (FIG. 27 ) and/or GPU(s), may provide signals (e.g.,representative of commands) to one or more components and/or systems ofthe autonomous vehicle 1502. For example, the controller(s) may sendsignals to operate the vehicle brakes via one or more brake actuators2510, to operate the steering system 2516 via one or more steeringactuators 2512, to operate the propulsion system 2526 via one or morethrottle/accelerator(s) 2538. The controller(s) 2544 may include one ormore onboard (e.g., integrated) computing devices (e.g., supercomputers)that process sensor signals, and output operation commands (e.g.,signals representing commands) to enable autonomous driving and/or toassist a human driver in driving the autonomous vehicle 1502. Thecontroller(s) 2544 may include a first controller for autonomous drivingfunctions, a second controller for functional safety functions, a thirdcontroller for artificial intelligence functionality (e.g., computervision), a fourth controller for infotainment functionality, a fifthcontroller for redundancy in emergency conditions, and/or othercontrollers. In some examples, a single controller may handle two ormore of the above functionalities, two or more controller(s) 2544 mayhandle a single functionality, and/or any combination thereof.

The controller(s) 2544 may provide the signals for controlling one ormore components and/or systems of the autonomous vehicle 1502 inresponse to sensor data received from one or more sensors (e.g., sensorinputs). The sensor data may be received from, for example and withoutlimitation, global navigation satellite systems sensors (GNSS sensor(s)2514) (e.g., Global Positioning System sensor(s)), RADAR sensor(s) 2528,ultrasonic sensor(s) 2542, LIDAR sensor(s) 2530, inertial measurementunit sensor(s) (IMU sensor(s) 2508) (e.g., accelerometer(s),gyroscope(s), magnetic compass(es), magnetometer(s), etc.),microphone(s) 2540, stereo camera(s) 2504, wide-view camera(s) 2506(e.g., fisheye cameras), infrared infrared camera(s) 2502, surroundcamera(s) 2518 (e.g., 360 degree cameras), long-range and/or mid-rangecamera(s) 2602, speed speed sensor(s) 2536 (e.g., for measuring thespeed of the autonomous vehicle 1502), vibration sensor(s) 2534,steering sensor(s) 2524, brake sensor(s) (e.g., as part of the brakesensor system 2532), and/or other sensor types.

One or more of the controller(s) 2544 may receive inputs (e.g.,represented by input data) from an instrument cluster 2520 of theautonomous vehicle 1502 and provide outputs (e.g., represented by outputdata, display data, etc.) via a human-machine interface display (HMIdisplay 2522), an audible annunciator, a loudspeaker, and/or via othercomponents of the autonomous vehicle 1502. The outputs may includeinformation such as vehicle velocity, speed, time, map data (e.g., theHD map 2708 of FIG. 27 ), location data (e.g., the autonomous vehicle1502 location, such as on a map), direction, location of other vehicles(e.g., an occupancy grid), information about objects and status ofobjects as perceived by the controller(s) 2544, etc. For example, theHMI display 2522 may display information about the presence of one ormore objects (e.g., a street sign, caution sign, traffic light changing,etc.), and/or information about driving maneuvers the vehicle has made,is making, or will make (e.g., changing lanes now, taking exit 34B intwo miles, etc.).

The autonomous vehicle 1502 further includes a network interface 2548which may use one or more wireless antenna(s) 2546 and/or modem(s) tocommunicate over one or more networks. For example, the networkinterface 2548 may be capable of communication over LTE, WCDMA, UMTS,GSM, CDMA2000, etc. The wireless antenna(s) 2546 may also enablecommunication between objects in the environment (e.g., vehicles, mobiledevices, etc.), using local area network(s), such as Bluetooth,Bluetooth LE, Z-Wave, ZigBee, etc., and/or low power wide-areanetwork(s) (LPWANs), such as LoRaWAN, SigFox, etc.

FIG. 26 is an example of camera locations and fields of view for theexample autonomous vehicle 1502 of FIG. 25 , in accordance with someembodiments of the present disclosure. The cameras and respective fieldsof view are one example embodiment and are not intended to be limiting.For example, additional and/or alternative cameras may be includedand/or the cameras may be located at different locations on theautonomous vehicle 1502.

The camera types for the cameras may include, but are not limited to,digital cameras that may be adapted for use with the components and/orsystems of the autonomous vehicle 1502. The camera(s) may operate atautomotive safety integrity level (ASIL) B and/or at another ASIL. Thecamera types may be capable of any image capture rate, such as 60 framesper second (fps), 720 fps, 240 fps, etc., depending on the embodiment.The cameras may be capable of using rolling shutters, global shutters,another type of shutter, or a combination thereof. In some examples, thecolor filter array may include a red clear clear clear (RCCC) colorfilter array, a red clear clear blue (RCCB) color filter array, a redblue green clear (RBGC) color filter array, a Foveon X3 color filterarray, a Bayer sensors (RGGB) color filter array, a monochrome sensorcolor filter array, and/or another type of color filter array. In someembodiments, clear pixel cameras, such as cameras with an RCCC, an RCCB,and/or an RBGC color filter array, may be used in an effort to increaselight sensitivity.

In some examples, one or more of the camera(s) may be used to performadvanced driver assistance systems (ADAS) functions (e.g., as part of aredundant or fail-safe design). For example, a Multi-Function MonoCamera may be installed to provide functions including lane departurewarning, traffic sign assist and intelligent headlamp control. One ormore of the camera(s) (e.g., all of the cameras) may record and provideimage data (e.g., video) simultaneously.

One or more of the cameras may be mounted in a mounting assembly, suchas a custom designed (3-D printed) assembly, in order to cut out straylight and reflections from within the car (e.g., reflections from thedashboard reflected in the windshield mirrors) which may interfere withthe camera's image data capture abilities. With reference to wing-mirrormounting assemblies, the wing-mirror assemblies may be custom 3-Dprinted so that the camera mounting plate matches the shape of thewing-mirror. In some examples, the camera(s) may be integrated into thewing-mirror. For side-view cameras, the camera(s) may also be integratedwithin the four pillars at each corner of the cabin.

Cameras with a field of view that include portions of the environment infront of the autonomous vehicle 1502 (e.g., front-facing cameras) may beused for surround view, to help identify forward facing paths andobstacles, as well aid in, with the help of one or more controller(s)2544 and/or control SoCs, providing information critical to generatingan occupancy grid and/or determining the preferred vehicle paths.Front-facing cameras may be used to perform many of the same ADASfunctions as LIDAR, including emergency braking, pedestrian detection,and collision avoidance. Front-facing cameras may also be used for ADASfunctions and systems including Lane Departure Warnings (“LDW”),Autonomous Cruise Control (“ACC”), and/or other functions such astraffic sign recognition.

A variety of cameras may be used in a front-facing configuration,including, for example, a monocular camera platform that includes a CMOS(complementary metal oxide semiconductor) color imager. Another examplemay be a wide-view camera(s) 2506 that may be used to perceive objectscoming into view from the periphery (e.g., pedestrians, crossing trafficor bicycles). Although only one wide-view camera is illustrated in FIG.26 , there may any number of wide-view camera(s) 2506 on the autonomousvehicle 1502. In addition, long-range camera(s) (long-range and/ormid-range camera(s) 2602) (e.g., a long-view stereo camera pair) may beused for depth-based object detection, especially for objects for whicha neural network has not yet been trained. The long-range camera(s) mayalso be used for object detection and classification, as well as basicobject tracking.

One or more stereo camera(s) 2504 may also be included in a front-facingconfiguration. The stereo camera(s) 2504 may include an integratedcontrol unit comprising a scalable processing unit, which may provide aprogrammable logic (FPGA) and a multi-core micro-processor with anintegrated CAN or Ethernet interface on a single chip. Such a unit maybe used to generate a 3-D map of the vehicle's environment, including adistance estimate for all the points in the image. Alternative stereocamera(s) 2504 may include a compact stereo vision sensor(s) that mayinclude two camera lenses (one each on the left and right) and an imageprocessing chip that may measure the distance from the vehicle to thetarget object and use the generated information (e.g., metadata) toactivate the autonomous emergency braking and lane departure warningfunctions. Other types of stereo camera(s) 2504 may be used in additionto, or alternatively from, those described herein.

Cameras with a field of view that include portions of the environment tothe side of the vehicle 102 (e.g., side-view cameras) may be used forsurround view, providing information used to create and update theoccupancy grid, as well as to generate side impact collision warnings.For example, surround camera(s) 2518 (e.g., four surround camera(s) 2518as illustrated in FIG. 26 ) may be positioned to on the vehicle 102. Thesurround camera(s) 2518 may include wide-view camera(s) 2506, fisheyecamera(s), 360 degree camera(s), and/or the like. Four example, fourfisheye cameras may be positioned on the vehicle's front, rear, andsides. In an alternative arrangement, the vehicle may use three surroundcamera(s) 2518 (e.g., left, right, and rear), and may leverage one ormore other camera(s) (e.g., a forward-facing camera) as a fourthsurround view camera.

Cameras with a field of view that include portions of the environment tothe rear of the autonomous vehicle 1502 (e.g., rear-view cameras) may beused for park assistance, surround view, rear collision warnings, andcreating and updating the occupancy grid. A wide variety of cameras maybe used including, but not limited to, cameras that are also suitable asa front-facing camera(s) (e.g., long-range and/or mid-range camera(s)2602, stereo camera(s) 2504, infrared camera(s) 2502, etc.), asdescribed herein.

FIG. 27 is a block diagram of an example system architecture for theexample autonomous vehicle 1502 of FIG. 25 , in accordance with someembodiments of the present disclosure. It should be understood that thisand other arrangements described herein are set forth only as examples.Other arrangements and elements (e.g., machines, interfaces, functions,orders, groupings of functions, etc.) may be used in addition to orinstead of those shown, and some elements may be omitted altogether.Further, many of the elements described herein are functional entitiesthat may be implemented as discrete or distributed components or inconjunction with other components, and in any suitable combination andlocation. Various functions described herein as being performed byentities may be carried out by hardware, firmware, and/or software. Forinstance, various functions may be carried out by a processor executinginstructions stored in memory.

Each of the components, features, and systems of the autonomous vehicle1502 in FIG. 27 are illustrated as being connected via bus 2728. The bus2728 may include a Controller Area Network (CAN) data interface(alternatively referred to herein as a “CAN bus”). A CAN may be anetwork inside the autonomous vehicle 1502 used to aid in control ofvarious features and functionality of the autonomous vehicle 1502, suchas actuation of brakes, acceleration, braking, steering, windshieldwipers, etc. A CAN bus may be configured to have dozens or even hundredsof nodes, each with its own unique identifier (e.g., a CAN ID). The CANbus may be read to find steering wheel angle, ground speed, enginerevolutions per minute (RPMs), button positions, and/or other vehiclestatus indicators. The CAN bus may be ASIL B compliant.

Although the bus 2728 is described herein as being a CAN bus, this isnot intended to be limiting. For example, in addition to, oralternatively from, the CAN bus, FlexRay and/or Ethernet may be used.Additionally, although a single line is used to represent the bus 2728,this is not intended to be limiting. For example, there may be anynumber of busses, which may include one or more CAN busses, one or moreFlexRay busses, one or more Ethernet busses, and/or one or more othertypes of busses using a different protocol. In some examples, two ormore busses may be used to perform different functions, and/or may beused for redundancy. For example, a first bus 2728 may be used forcollision avoidance functionality and a second bus 2728 may be used foractuation control. In any example, each bus 2728 may communicate withany of the components of the autonomous vehicle 1502, and two or morebusses may communicate with the same components. In some examples, eachSoC(s) 2702, each of the controller(s) 2544, and/or each computer withinthe vehicle may have access to the same input data (e.g., inputs fromsensors of the autonomous vehicle 1502), and may be connected to acommon bus, such the CAN bus.

The autonomous vehicle 1502 may include one or more controller(s) 2544,such as those described herein with respect to FIG. 25 . Thecontroller(s) 2544 may be used for a variety of functions. Thecontroller(s) 2544 may be coupled to any of the various other componentsand systems of the autonomous vehicle 1502, and may be used for controlof the autonomous vehicle 1502, artificial intelligence of theautonomous vehicle 1502, infotainment for the autonomous vehicle 1502,and/or the like.

The autonomous vehicle 1502 may include SoC(s) 2702. The SoC(s) 2702 mayinclude CPU(s) 2724, GPU(s) 2726, processor(s) 2722, cache(s) 2720,accelerator(s) 2718, data store(s) 2714, and/or other components andfeatures not illustrated. The SoC(s) 2702 may be used to control theautonomous vehicle 1502 in a variety of platforms and systems. Forexample, the SoC(s) 2702 may be combined in a system (e.g., the systemof the vehicle 102) with an HD map 2708 which may obtain map refreshesand/or updates via a network interface 2548 from one or more servers(e.g., server(s) 2802 of FIG. 28 ).

The CPU(s) 2724 may include a CPU cluster or CPU complex (alternativelyreferred to herein as a “CCPLEX”). The CPU(s) 2724 may include multiplecores and/or L2 caches. For example, in some embodiments, the CPU(s)2724 may include eight cores in a coherent multiprocessor configuration.In some embodiments, the CPU(s) 2724 may include four dual-core clusterswhere each cluster has a dedicated L2 cache (e.g., a 2 MB L2 cache). TheCPU(s) 2724 (e.g., the CCPLEX) may be configured to support simultaneouscluster operation enabling any combination of the clusters of the CPU(s)2724 to be active at any given time.

The CPU(s) 2724 may implement power management capabilities that includeone or more of the following features: individual hardware blocks may beclock-gated automatically when idle to save dynamic power; each coreclock may be gated when the core is not actively executing instructionsdue to execution of WFI/WFE instructions; each core may be independentlypower-gated; each core cluster may be independently clock-gated when allcores are clock-gated or power-gated; and/or each core cluster may beindependently power-gated when all cores are power-gated. The CPU(s)2724 may further implement an enhanced algorithm for managing powerstates, where allowed power states and expected wakeup times arespecified, and the hardware/microcode determines the best power state toenter for the core, cluster, and CCPLEX. The processing cores maysupport simplified power state entry sequences in software with the workoffloaded to microcode.

The GPU(s) 2726 may include an integrated GPU (alternatively referred toherein as an “iGPU”). The GPU(s) 2726 may be programmable and may beefficient for parallel workloads. The GPU(s) 2726, in some examples, mayuse an enhanced tensor instruction set. The GPU(s) 2726 may include oneor more streaming microprocessors, where each streaming microprocessormay include an L1 cache (e.g., an L1 cache with at least 96 KB storagecapacity), and two or more of the streaming microprocessors may share anL2 cache (e.g., an L2 cache with a 512 KB storage capacity). In someembodiments, the GPU(s) 2726 may include at least eight streamingmicroprocessors. The GPU(s) 2726 may use compute application programminginterface(s) (API(s)). In addition, the GPU(s) 2726 may use one or moreparallel computing platforms and/or programming models (e.g., NVIDIA'sCUDA).

The GPU(s) 2726 may be power-optimized for best performance inautomotive and embedded use cases. For example, the GPU(s) 2726 may befabricated on a Fin field-effect transistor (FinFET). However, this isnot intended to be limiting and the GPU(s) 2726 may be fabricated usingother semiconductor manufacturing processes. Each streamingmicroprocessor may incorporate a number of mixed-precision processingcores partitioned into multiple blocks. For example, and withoutlimitation, 64 PF32 cores and 32 PF64 cores may be partitioned into fourprocessing blocks. In such an example, each processing block may beallocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, twomixed-precision NVIDIA TENSOR COREs for deep learning matrix arithmetic,an L0 instruction cache, a warp scheduler, a dispatch unit, and/or a 64KB register file. In addition, the streaming microprocessors may includeindependent parallel integer and floating-point data paths to providefor efficient execution of workloads with a mix of computation andaddressing calculations. The streaming microprocessors may includeindependent thread scheduling capability to enable finer-grainsynchronization and cooperation between parallel threads. The streamingmicroprocessors may include a combined L1 data cache and shared memoryunit in order to improve performance while simplifying programming.

The GPU(s) 2726 may include a high bandwidth memory (HBM) and/or a 16 GBHBM2 memory subsystem to provide, in some examples, about 900 GB/secondpeak memory bandwidth. In some examples, in addition to, oralternatively from, the HBM memory, a synchronous graphics random-accessmemory (SGRAM) may be used, such as a graphics double data rate typefive synchronous random-access memory (GDDRS).

The GPU(s) 2726 may include unified memory technology including accesscounters to allow for more accurate migration of memory pages to theprocessor that accesses them most frequently, thereby improvingefficiency for memory ranges shared between processors. In someexamples, address translation services (ATS) support may be used toallow the GPU(s) 2726 to access the CPU(s) 2724 page tables directly. Insuch examples, when the GPU(s) 2726 memory management unit (MMU)experiences a miss, an address translation request may be transmitted tothe CPU(s) 2724. In response, the CPU(s) 2724 may look in its pagetables for the virtual-to-physical mapping for the address and transmitsthe translation back to the GPU(s) 2726. As such, unified memorytechnology may allow a single unified virtual address space for memoryof both the CPU(s) 2724 and the GPU(s) 2726, thereby simplifying theGPU(s) 2726 programming and porting of applications to the GPU(s) 2726.

In addition, the GPU(s) 2726 may include an access counter that may keeptrack of the frequency of access of the GPU(s) 2726 to memory of otherprocessors. The access counter may help ensure that memory pages aremoved to the physical memory of the processor that is accessing thepages most frequently.

The SoC(s) 2702 may include any number of cache(s) 2720, including thosedescribed herein. For example, the cache(s) 2720 may include an L3 cachethat is available to both the CPU(s) 2724 and the GPU(s) 2726 (e.g.,that is connected both the CPU(s) 2724 and the GPU(s) 2726). Thecache(s) 2720 may include a write-back cache that may keep track ofstates of lines, such as by using a cache coherence protocol (e.g., MEI,MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending onthe embodiment, although smaller cache sizes may be used.

The SoC(s) 2702 may include one or more accelerator(s) 2718 (e.g.,hardware accelerators, software accelerators, or a combination thereof).For example, the SoC(s) 2702 may include a hardware acceleration clusterthat may include optimized hardware accelerators and/or large on-chipmemory. The large on-chip memory (e.g., 4 MB of SRAM), may enable thehardware acceleration cluster to accelerate neural networks and othercalculations. The hardware acceleration cluster may be used tocomplement the GPU(s) 2726 and to offload some of the tasks of theGPU(s) 2726 (e.g., to free up more cycles of the GPU(s) 2726 forperforming other tasks). As an example, the accelerator(s) 2718 may beused for targeted workloads (e.g., perception, convolutional neuralnetworks (CNNs), etc.) that are stable enough to be amenable toacceleration. The term “CNN,” as used herein, may include all types ofCNNs, including region-based or regional convolutional neural networks(RCNNs) and Fast RCNNs (e.g., as used for object detection).

The accelerator(s) 2718 (e.g., the hardware acceleration cluster) mayinclude a deep learning accelerator(s) (DLA). The DLA(s) may include oneor more Tensor processing units (TPUs) that may be configured to providean additional ten trillion operations per second for deep learningapplications and inferencing. The TPUs may be accelerators configuredto, and optimized for, performing image processing functions (e.g., forCNNs, RCNNs, etc.). The DLA(s) may further be optimized for a specificset of neural network types and floating point operations, as well asinferencing. The design of the DLA(s) may provide more performance permillimeter than a general-purpose GPU, and vastly exceeds theperformance of a CPU. The TPU(s) may perform several functions,including a single-instance convolution function, supporting, forexample, INT8, INT16, and FP16 data types for both features and weights,as well as post-processor functions.

The DLA(s) may quickly and efficiently execute neural networks,especially CNNs, on processed or unprocessed data for any of a varietyof functions, including, for example and without limitation: a CNN forobject identification and detection using data from camera sensors; aCNN for distance estimation using data from camera sensors; a CNN foremergency vehicle detection and identification and detection using datafrom microphones; a CNN for facial recognition and vehicle owneridentification using data from camera sensors; and/or a CNN for securityand/or safety related events.

The DLA(s) may perform any function of the GPU(s) 2726, and by using aninference accelerator, for example, a designer may target either theDLA(s) or the GPU(s) 2726 for any function. For example, the designermay focus processing of CNNs and floating point operations on the DLA(s)and leave other functions to the GPU(s) 2726 and/or other accelerator(s)2718.

The accelerator(s) 2718 (e.g., the hardware acceleration cluster) mayinclude a programmable vision accelerator(s) (PVA), which mayalternatively be referred to herein as a computer vision accelerator.The PVA(s) may be designed and configured to accelerate computer visionalgorithms for the advanced driver assistance systems (ADAS), autonomousdriving, and/or augmented reality (AR) and/or virtual reality (VR)applications. The PVA(s) may provide a balance between performance andflexibility. For example, each PVA(s) may include, for example andwithout limitation, any number of reduced instruction set computer(RISC) cores, direct memory access (DMA), and/or any number of vectorprocessors.

The RISC cores may interact with image sensors (e.g., the image sensorsof any of the cameras described herein), image signal processor(s),and/or the like. Each of the RISC cores may include any amount ofmemory. The RISC cores may use any of a number of protocols, dependingon the embodiment. In some examples, the RISC cores may execute areal-time operating system (RTOS). The RISC cores may be implementedusing one or more integrated circuit devices, application specificintegrated circuits (ASICs), and/or memory devices. For example, theRISC cores may include an instruction cache and/or a tightly coupledRAM.

The DMA may enable components of the PVA(s) to access the system memoryindependently of the CPU(s) 2724. The DMA may support any number offeatures used to provide optimization to the PVA including, but notlimited to, supporting multi-dimensional addressing and/or circularaddressing. In some examples, the DMA may support up to six or moredimensions of addressing, which may include block width, block height,block depth, horizontal block stepping, vertical block stepping, and/ordepth stepping.

The vector processors may be programmable processors that may bedesigned to efficiently and flexibly execute programming for computervision algorithms and provide signal processing capabilities. In someexamples, the PVA may include a PVA core and two vector processingsubsystem partitions. The PVA core may include a processor subsystem,DMA engine(s) (e.g., two DMA engines), and/or other peripherals. Thevector processing subsystem may operate as the primary processing engineof the PVA, and may include a vector processing unit (VPU), aninstruction cache, and/or vector memory (e.g., VMEM). A VPU core mayinclude a digital signal processor such as, for example, a singleinstruction, multiple data (SIMD), very long instruction word (VLIW)digital signal processor. The combination of the SIMD and VLIW mayenhance throughput and speed.

Each of the vector processors may include an instruction cache and maybe coupled to dedicated memory. As a result, in some examples, each ofthe vector processors may be configured to execute independently of theother vector processors. In other examples, the vector processors thatare included in a particular PVA may be configured to employ dataparallelism. For example, in some embodiments, the plurality of vectorprocessors included in a single PVA may execute the same computer visionalgorithm, but on different regions of an image. In other examples, thevector processors included in a particular PVA may simultaneouslyexecute different computer vision algorithms, on the same image, or evenexecute different algorithms on sequential images or portions of animage. Among other things, any number of PVAs may be included in thehardware acceleration cluster and any number of vector processors may beincluded in each of the PVAs. In addition, the PVA(s) may includeadditional error correcting code (ECC) memory, to enhance overall systemsafety.

The accelerator(s) 2718 (e.g., the hardware acceleration cluster) mayinclude a computer vision network on-chip and SRAM, for providing ahigh-bandwidth, low latency SRAM for the accelerator(s) 2718. In someexamples, the on-chip memory may include at least 4 MB SRAM, consistingof, for example and without limitation, eight field-configurable memoryblocks, that may be accessible by both the PVA and the DLA. Each pair ofmemory blocks may include an advanced peripheral bus (APB) interface,configuration circuitry, a controller, and a multiplexer. Any type ofmemory may be used. The PVA and DLA may access the memory via a backbonethat provides the PVA and DLA with high-speed access to memory. Thebackbone may include a computer vision network on-chip thatinterconnects the PVA and the DLA to the memory (e.g., using the APB).

The computer vision network on-chip may include an interface thatdetermines, before transmission of any control signal/address/data, thatboth the PVA and the DLA provide ready and valid signals. Such aninterface may provide for separate phases and separate channels fortransmitting control signals/addresses/data, as well as burst-typecommunications for continuous data transfer. This type of interface maycomply with ISO 26262 or IEC 61506 standards, although other standardsand protocols may be used.

In some examples, the SoC(s) 2702 may include a real-time ray-tracinghardware accelerator, such as described in U.S. patent application Ser.No. 16/101,232, filed on Aug. 10, 2018. The real-time ray-tracinghardware accelerator may be used to quickly and efficiently determinethe positions and extents of objects (e.g., within a world model), togenerate realtime visualization simulations, for RADAR signalinterpretation, for sound propagation synthesis and/or analysis, forsimulation of SONAR systems, for general wave propagation simulation,for comparison to LIDAR data for purposes of localization and/or otherfunctions, and/or for other uses.

The accelerator(s) 2718 (e.g., the hardware accelerator cluster) have awide array of uses for autonomous driving. The PVA may be a programmablevision accelerator that may be used for key processing stages in ADASand autonomous vehicles. The PVA's capabilities are a good match foralgorithmic domains needing predictable processing, at low power and lowlatency. In other words, the PVA performs well on semi-dense or denseregular computation, even on small data sets, which need predictablerun-times with low latency and low power. Thus, in the context ofplatforms for autonomous vehicles, the PVAs are designed to run classiccomputer vision algorithms, as they are efficient at object detectionand operating on integer math.

For example, according to one embodiment of the technology, the PVA isused to perform computer stereo vision. A semi-global matching-basedalgorithm may be used in some examples, although this is not intended tobe limiting. Many applications for Level 3-5 autonomous driving requiremotion estimation/stereo matching on-the-fly (e.g., structure frommotion, pedestrian recognition, lane detection, etc.). The PVA mayperform computer stereo vision function on inputs from two monocularcameras.

In some examples, the PVA may be used to perform dense optical flow.According to process raw RADAR data (e.g., using a 4D Fast FourierTransform) to provide Processed RADAR. In other examples, the PVA isused for time of flight depth processing, by processing raw time offlight data to provide processed time of flight data, for example.

The DLA may be used to run any type of network to enhance control anddriving safety, including for example, a neural network that outputs ameasure of confidence for each object detection. Such a confidence valuemay be interpreted as a probability, or as providing a relative “weight”of each detection compared to other detections. This confidence valueenables the system to make further decisions regarding which detectionsshould be considered as true positive detections rather than falsepositive detections. For example, the system may set a threshold valuefor the confidence and consider only the detections exceeding thethreshold value as true positive detections. In an automatic emergencybraking (AEB) system, false positive detections would cause the vehicleto automatically perform emergency braking, which is obviouslyundesirable. Therefore, only the most confident detections should beconsidered as triggers for AEB. The DLA may run a neural network forregressing the confidence value. The neural network may take as itsinput at least some subset of parameters, such as bounding boxdimensions, ground plane estimate obtained (e.g. from anothersubsystem), inertial measurement unit sensor(s) (IMU sensor(s) 2508)output that correlates with the autonomous vehicle 1502 orientation,distance, 3D location estimates of the object obtained from the neuralnetwork and/or other sensors (e.g., LIDAR sensor(s) 2530 or RADARsensor(s) 2528), among others.

The SoC(s) 2702 may include data store(s) 2714 (e.g., memory). The datastore(s) 2714 may be on-chip memory of the SoC(s) 2702, which may storeneural networks to be executed on the GPU and/or the DLA. In someexamples, the data store(s) 2714 may be large enough in capacity tostore multiple instances of neural networks for redundancy and safety.The data store(s) 2714 may comprise L2 or L3 cache(s) 2720. Reference tothe data store(s) 2714 may include reference to the memory associatedwith the PVA, DLA, and/or other accelerator(s) 2718, as describedherein.

The SoC(s) 2702 may include one or more processor(s) 2722 (e.g.,embedded processors). The processor(s) 2722 may include a boot and powermanagement processor that may be a dedicated processor and subsystem tohandle boot power and management functions and related securityenforcement. The boot and power management processor may be a part ofthe SoC(s) 2702 boot sequence and may provide runtime power managementservices. The boot power and management processor may provide clock andvoltage programming, assistance in system low power state transitions,management of SoC(s) 2702 thermals and temperature sensors, and/ormanagement of the SoC(s) 2702 power states. Each temperature sensor maybe implemented as a ring-oscillator whose output frequency isproportional to temperature, and the SoC(s) 2702 may use thering-oscillators to detect temperatures of the CPU(s) 2724, GPU(s) 2726,and/or accelerator(s) 2718. If temperatures are determined to exceed athreshold, the boot and power management processor may enter atemperature fault routine and put the SoC(s) 2702 into a lower powerstate and/or put the autonomous vehicle 1502 into a chauffeur to safestop mode (e.g., bring the autonomous vehicle 1502 to a safe stop).

The processor(s) 2722 may further include a set of embedded processorsthat may serve as an audio processing engine. The audio processingengine may be an audio subsystem that enables full hardware support formulti-channel audio over multiple interfaces, and a broad and flexiblerange of audio I/O interfaces. In some examples, the audio processingengine is a dedicated processor core with a digital signal processorwith dedicated RAM.

The processor(s) 2722 may further include an always on processor enginethat may provide necessary hardware features to support low power sensormanagement and wake use cases. The always on processor engine mayinclude a processor core, a tightly coupled RAM, supporting peripherals(e.g., timers and interrupt controllers), various I/O controllerperipherals, and routing logic.

The processor(s) 2722 may further include a safety cluster engine thatincludes a dedicated processor subsystem to handle safety management forautomotive applications. The safety cluster engine may include two ormore processor cores, a tightly coupled RAM, support peripherals (e.g.,timers, an interrupt controller, etc.), and/or routing logic. In asafety mode, the two or more cores may operate in a lockstep mode andfunction as a single core with comparison logic to detect anydifferences between their operations.

The processor(s) 2722 may further include a real-time camera engine thatmay include a dedicated processor subsystem for handling real-timecamera management.

The processor(s) 2722 may further include a high-dynamic range signalprocessor that may include an image signal processor that is a hardwareengine that is part of the camera processing pipeline.

The processor(s) 2722 may include a video image compositor that may be aprocessing block (e.g., implemented on a microprocessor) that implementsvideo post-processing functions needed by a video playback applicationto produce the final image for the player window. The video imagecompositor may perform lens distortion correction on wide-view camera(s)2506, surround camera(s) 2518, and/or on in-cabin monitoring camerasensors. In-cabin monitoring camera sensor is preferably monitored by aneural network running on another instance of the Advanced SoC,configured to identify in cabin events and respond accordingly. Anin-cabin system may perform lip reading to activate cellular service andplace a phone call, dictate emails, change the vehicle's destination,activate or change the vehicle's infotainment system and settings, orprovide voice-activated web surfing. Certain functions are available tothe driver only when the vehicle is operating in an autonomous mode, andare disabled otherwise.

The video image compositor may include enhanced temporal noise reductionfor both spatial and temporal noise reduction. For example, where motionoccurs in a video, the noise reduction weights spatial informationappropriately, decreasing the weight of information provided by adjacentframes. Where an image or portion of an image does not include motion,the temporal noise reduction performed by the video image compositor mayuse information from the previous image to reduce noise in the currentimage.

The video image compositor may also be configured to perform stereorectification on input stereo lens frames. The video image compositormay further be used for user interface composition when the operatingsystem desktop is in use, and the GPU(s) 2726 is not required tocontinuously render new surfaces. Even when the GPU(s) 2726 is poweredon and active doing 3D rendering, the video image compositor may be usedto offload the GPU(s) 2726 to improve performance and responsiveness.

The SoC(s) 2702 may further include a mobile industry processorinterface (MIPI) camera serial interface for receiving video and inputfrom cameras, a high-speed interface, and/or a video input block thatmay be used for camera and related pixel input functions. The SoC(s)2702 may further include an input/output controller(s) that may becontrolled by software and may be used for receiving I/O signals thatare uncommitted to a specific role.

The SoC(s) 2702 may further include a broad range of peripheralinterfaces to enable communication with peripherals, audio codecs, powermanagement, and/or other devices. The SoC(s) 2702 may be used to processdata from cameras (e.g., connected over Gigabit Multimedia Serial Linkand Ethernet), sensors (e.g., LIDAR sensor(s) 2530, RADAR sensor(s)2528, etc. that may be connected over Ethernet), data from bus 2728(e.g., speed of autonomous vehicle 1502, steering wheel position, etc.),data from GNSS sensor(s) 2514 (e.g., connected over Ethernet or CANbus). The SoC(s) 2702 may further include dedicated high-performancemass storage controllers that may include their own DMA engines, andthat may be used to free the CPU(s) 2724 from routine data managementtasks.

The SoC(s) 2702 may be an end-to-end platform with a flexiblearchitecture that spans automation levels 3-5, thereby providing acomprehensive functional safety architecture that leverages and makesefficient use of computer vision and ADAS techniques for diversity andredundancy, provides a platform for a flexible, reliable drivingsoftware stack, along with deep learning tools. The SoC(s) 2702 may befaster, more reliable, and even more energy-efficient andspace-efficient than conventional systems. For example, theaccelerator(s) 2718, when combined with the CPU(s) 2724, the GPU(s)2726, and the data store(s) 2714, may provide for a fast, efficientplatform for level 3-5 autonomous vehicles.

The technology thus provides capabilities and functionality that cannotbe achieved by conventional systems. For example, computer visionalgorithms may be executed on CPUs, which may be configured usinghigh-level programming language, such as the C programming language, toexecute a wide variety of processing algorithms across a wide variety ofvisual data. However, CPUs are oftentimes unable to meet the performancerequirements of many computer vision applications, such as those relatedto execution time and power consumption, for example. In particular,many CPUs are unable to execute complex object detection algorithms inreal-time, which is a requirement of in-vehicle ADAS applications, and arequirement for practical Level 3-5 autonomous vehicles.

In contrast to conventional systems, by providing a CPU complex, GPUcomplex, and a hardware acceleration cluster, the technology describedherein allows for multiple neural networks to be performedsimultaneously and/or sequentially, and for the results to be combinedtogether to enable Level 3-5 autonomous driving functionality. Forexample, a CNN executing on the DLA or dGPU (e.g., the GPU(s) 2712) mayinclude a text and word recognition, allowing the supercomputer to readand understand traffic signs, including signs for which the neuralnetwork has not been specifically trained. The DLA may further include aneural network that is able to identify, interpret, and providessemantic understanding of the sign, and to pass that semanticunderstanding to the path planning modules running on the CPU Complex.

As another example, multiple neural networks may be run simultaneously,as is required for Level 3, 4, or 5 driving. For example, a warning signconsisting of “Caution: flashing lights indicate icy conditions,” alongwith an electric light, may be independently or collectively interpretedby several neural networks. The sign itself may be identified as atraffic sign by a first deployed neural network (e.g., a neural networkthat has been trained), the text “Flashing lights indicate icyconditions” may be interpreted by a second deployed neural network,which informs the vehicle's path planning software (preferably executingon the CPU Complex) that when flashing lights are detected, icyconditions exist. The flashing light may be identified by operating athird deployed neural network over multiple frames, informing thevehicle's path-planning software of the presence (or absence) offlashing lights. All three neural networks may run simultaneously, suchas within the DLA and/or on the GPU(s) 2726.

In some examples, a CNN for facial recognition and vehicle owneridentification may use data from camera sensors to identify the presenceof an authorized driver and/or owner of the autonomous vehicle 1502. Thealways on sensor processing engine may be used to unlock the vehiclewhen the owner approaches the driver door and turn on the lights, and,in security mode, to disable the vehicle when the owner leaves thevehicle. In this way, the SoC(s) 2702 provide for security against theftand/or carjacking.

In another example, a CNN for emergency vehicle detection andidentification may use data from microphone(s) 2540 to detect andidentify emergency vehicle sirens. In contrast to conventional systems,that use general classifiers to detect sirens and manually extractfeatures, the SoC(s) 2702 use the CNN for classifying environmental andurban sounds, as well as classifying visual data. In a preferredembodiment, the CNN running on the DLA is trained to identify therelative closing speed of the emergency vehicle (e.g., by using theDoppler effect). The CNN may also be trained to identify emergencyvehicles specific to the local area in which the vehicle is operating,as identified by GNSS sensor(s) 2514. Thus, for example, when operatingin Europe the CNN will seek to detect European sirens, and when in theUnited States the CNN will seek to identify only North American sirens.Once an emergency vehicle is detected, a control program may be used toexecute an emergency vehicle safety routine, slowing the vehicle,pulling over to the side of the road, parking the vehicle, and/or idlingthe vehicle, with the assistance of ultrasonic sensor(s) 2542, until theemergency vehicle(s) passes.

The vehicle may include a CPU(s) 2710 (e.g., discrete CPU(s), ordCPU(s)), that may be coupled to the SoC(s) 2702 via a high-speedinterconnect (e.g., PCIe). The CPU(s) 2710 may include an X86 processor,for example. The CPU(s) 2710 may be used to perform any of a variety offunctions, including arbitrating potentially inconsistent resultsbetween ADAS sensors and the SoC(s) 2702, and/or monitoring the statusand health of the controller(s) 2544 and/or infotainment SoC 2704, forexample.

The autonomous vehicle 1502 may include a GPU(s) 2712 (e.g., discreteGPU(s), or dGPU(s)), that may be coupled to the SoC(s) 2702 via ahigh-speed interconnect (e.g., NVIDIA's NVLINK). The GPU(s) 2712 mayprovide additional artificial intelligence functionality, such as byexecuting redundant and/or different neural networks, and may be used totrain and/or update neural networks based on input (e.g., sensor data)from sensors of the autonomous vehicle 1502.

The autonomous vehicle 1502 may further include the network interface2548 which may include one or more wireless antenna(s) 2546 (e.g., oneor more wireless antennas for different communication protocols, such asa cellular antenna, a Bluetooth antenna, etc.). The network interface2548 may be used to enable wireless connectivity over the Internet withthe cloud (e.g., with the server(s) 2802 and/or other network devices),with other vehicles, and/or with computing devices (e.g., client devicesof passengers). To communicate with other vehicles, a direct link may beestablished between the two vehicles and/or an indirect link may beestablished (e.g., across networks and over the Internet). Direct linksmay be provided using a vehicle-to-vehicle communication link. Thevehicle-to-vehicle communication link may provide the autonomous vehicle1502 information about vehicles in proximity to the autonomous vehicle1502 (e.g., vehicles in front of, on the side of, and/or behind theautonomous vehicle 1502). This functionality may be part of acooperative adaptive cruise control functionality of the autonomousvehicle 1502.

The network interface 2548 may include a SoC that provides modulationand demodulation functionality and enables the controller(s) 2544 tocommunicate over wireless networks. The network interface 2548 mayinclude a radio frequency front-end for up-conversion from baseband toradio frequency, and down conversion from radio frequency to baseband.The frequency conversions may be performed through well-known processes,and/or may be performed using super-heterodyne processes. In someexamples, the radio frequency front end functionality may be provided bya separate chip. The network interface may include wirelessfunctionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000,Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and/or otherwireless protocols.

The autonomous vehicle 1502 may further include data store(s) 2716 whichmay include off-chip (e.g., off the SoC(s) 2702) storage. The datastore(s) 2716 may include one or more storage elements including RAM,SRAM, DRAM, VRAM, Flash, hard disks, and/or other components and/ordevices that may store at least one bit of data.

The autonomous vehicle 1502 may further include GNSS sensor(s) 2514. TheGNSS sensor(s) 2514 (e.g., GPS and/or assisted GPS sensors), to assistin mapping, perception, occupancy grid generation, and/or path planningfunctions. Any number of GNSS sensor(s) 2514 may be used, including, forexample and without limitation, a GPS using a USB connector with anEthernet to Serial (RS-232) bridge.

The vehicle 102 may further include RADAR sensor(s) 2528. The RADARsensor(s) 2528 may be used by the vehicle 102 for long-range vehicledetection, even in darkness and/or severe weather conditions. RADARfunctional safety levels may be ASIL B. The RADAR sensor(s) 2528 may usethe CAN and/or the bus 2728 (e.g., to transmit data generated by theRADAR sensor(s) 2528) for control and to access object tracking data,with access to Ethernet to access raw data in some examples. A widevariety of RADAR sensor types may be used. For example, and withoutlimitation, the RADAR sensor(s) 2528 may be suitable for front, rear,and side RADAR use. In some example, Pulse Doppler RADAR sensor(s) areused.

The RADAR sensor(s) 2528 may include different configurations, such aslong range with narrow field of view, short range with wide field ofview, short range side coverage, etc. In some examples, long-range RADARmay be used for adaptive cruise control functionality. The long-rangeRADAR systems may provide a broad field of view realized by two or moreindependent scans, such as within a 250 m range. The RADAR sensor(s)2528 may help in distinguishing between static and moving objects, andmay be used by ADAS systems for emergency brake assist and forwardcollision warning. Long-range RADAR sensors may include monostaticmultimodal RADAR with multiple (e.g., six or more) fixed RADAR antennaeand a high-speed CAN and FlexRay interface. In an example with sixantennae, the central four antennae may create a focused beam pattern,designed to record the vehicle's surroundings at higher speeds withminimal interference from traffic in adjacent lanes. The other twoantennae may expand the field of view, making it possible to quicklydetect vehicles entering or leaving the vehicle's lane.

Mid-range RADAR systems may include, as an example, a range of up to 760m (front) or 80 m (rear), and a field of view of up to 42 degrees(front) or 750 degrees (rear). Short-range RADAR systems may include,without limitation, RADAR sensors designed to be installed at both endsof the rear bumper. When installed at both ends of the rear bumper, sucha RADAR sensor systems may create two beams that constantly monitor theblind spot in the rear and next to the vehicle.

Short-range RADAR systems may be used in an ADAS system for blind spotdetection and/or lane change assist.

The autonomous vehicle 1502 may further include ultrasonic sensor(s)2542. The ultrasonic sensor(s) 2542, which may be positioned at thefront, back, and/or the sides of the autonomous vehicle 1502, may beused for park assist and/or to create and update an occupancy grid. Awide variety of ultrasonic sensor(s) 2542 may be used, and differentultrasonic sensor(s) 2542 may be used for different ranges of detection(e.g., 2.5 m, 4 m). The ultrasonic sensor(s) 2542 may operate atfunctional safety levels of ASIL B.

The autonomous vehicle 1502 may include LIDAR sensor(s) 2530. The LIDARsensor(s) 2530 may be used for object and pedestrian detection,emergency braking, collision avoidance, and/or other functions. TheLIDAR sensor(s) 2530 may be functional safety level ASIL B. In someexamples, the autonomous vehicle 1502 may include multiple LIDARsensor(s) 2530 (e.g., two, four, six, etc.) that may use Ethernet (e.g.,to provide data to a Gigabit Ethernet switch).

In some examples, the LIDAR sensor(s) 2530 may be capable of providing alist of objects and their distances for a 360-degree field of view.Commercially available LIDAR sensor(s) 2530 may have an advertised rangeof approximately 102 m, with an accuracy of 2 cm-3 cm, and with supportfor a 102 Mbps Ethernet connection, for example. In some examples, oneor more non-protruding LIDAR sensor(s) 2530 may be used. In suchexamples, the LIDAR sensor(s) 2530 may be implemented as a small devicethat may be embedded into the front, rear, sides, and/or corners of theautonomous vehicle 1502. The LIDAR sensor(s) 2530, in such examples, mayprovide up to a 720-degree horizontal and 35-degree verticalfield-of-view, with a 200 m range even for low-reflectivity objects.Front-mounted LIDAR sensor(s) 2530 may be configured for a horizontalfield of view between 45 degrees and 135 degrees.

In some examples, LIDAR technologies, such as 3D flash LIDAR, may alsobe used. 3D Flash LIDAR uses a flash of a laser as a transmissionsource, to illuminate vehicle surroundings up to approximately 200 m. Aflash LIDAR unit includes a receptor, which records the laser pulsetransit time and the reflected light on each pixel, which in turncorresponds to the range from the vehicle to the objects. Flash LIDARmay allow for highly accurate and distortion-free images of thesurroundings to be generated with every laser flash. In some examples,four flash LIDAR sensors may be deployed, one at each side of theautonomous vehicle 1502. Available 3D flash LIDAR systems include asolid-state 3D staring array LIDAR camera with no moving parts otherthan a fan (e.g., a non-scanning LIDAR device). The flash LIDAR devicemay use a 5 nanosecond class I (eye-safe) laser pulse per frame and maycapture the reflected laser light in the form of 3D range point cloudsand co-registered intensity data. By using flash LIDAR, and becauseflash LIDAR is a solid-state device with no moving parts, the LIDARsensor(s) 2530 may be less susceptible to motion blur, vibration, and/orshock.

The vehicle may further include IMU sensor(s) 2508. The IMU sensor(s)2508 may be located at a center of the rear axle of the autonomousvehicle 1502, in some examples. The IMU sensor(s) 2508 may include, forexample and without limitation, an accelerometer(s), a magnetometer(s),a gyroscope(s), a magnetic compass(es), and/or other sensor types. Insome examples, such as in six-axis applications, the IMU sensor(s) 2508may include accelerometers and gyroscopes, while in nine-axisapplications, the IMU sensor(s) 2508 may include accelerometers,gyroscopes, and magnetometers.

In some embodiments, the IMU sensor(s) 2508 may be implemented as aminiature, high performance GPS-Aided Inertial Navigation System(GPS/INS) that combines micro-electro-mechanical systems (MEMS) inertialsensors, a high-sensitivity GPS receiver, and advanced Kalman filteringalgorithms to provide estimates of position, velocity, and attitude. Assuch, in some examples, the IMU sensor(s) 2508 may enable the autonomousvehicle 1502 to estimate heading without requiring input from a magneticsensor by directly observing and correlating the changes in velocityfrom GPS to the IMU sensor(s) 2508. In some examples, the IMU sensor(s)2508 and the GNSS sensor(s) 2514 may be combined in a single integratedunit.

The vehicle may include microphone(s) 2540 placed in and/or around theautonomous vehicle 1502. The microphone(s) 2540 may be used foremergency vehicle detection and identification, among other things.

The vehicle may further include any number of camera types, includingstereo camera(s) 2504, wide-view camera(s) 2506, infrared camera(s)2502, surround camera(s) 2518, long-range and/or mid-range camera(s)2602, and/or other camera types. The cameras may be used to captureimage data around an entire periphery of the autonomous vehicle 1502.The types of cameras used depends on the embodiments and requirementsfor the autonomous vehicle 1502, and any combination of camera types maybe used to provide the necessary coverage around the autonomous vehicle1502. In addition, the number of cameras may differ depending on theembodiment. For example, the vehicle may include six cameras, sevencameras, ten cameras, twelve cameras, and/or another number of cameras.The cameras may support, as an example and without limitation, GigabitMultimedia Serial Link (GMSL) and/or Gigabit Ethernet. Each of thecamera(s) is described with more detail herein with respect to FIG. 25and FIG. 26 .

The autonomous vehicle 1502 may further include vibration sensor(s)2534. The vibration sensor(s) 2534 may measure vibrations of componentsof the vehicle, such as the axle(s). For example, changes in vibrationsmay indicate a change in road surfaces. In another example, when two ormore vibration sensor(s) 2534 are used, the differences between thevibrations may be used to determine friction or slippage of the roadsurface (e.g., when the difference in vibration is between apower-driven axle and a freely rotating axle).

The autonomous vehicle 1502 may include an ADAS system 2706. The ADASsystem 2706 may include a SoC, in some examples. The ADAS system 2706may include autonomous/adaptive/automatic cruise control (ACC),cooperative adaptive cruise control (CACC), forward crash warning (FCW),automatic emergency braking (AEB), lane departure warnings (LDW), lanekeep assist (LKA), blind spot warning (BSW), rear cross-traffic warning(RCTW), collision warning systems (CWS), lane centering (LC), and/orother features and functionality.

The ACC systems may use RADAR sensor(s) 2528, LIDAR sensor(s) 2530,and/or a camera(s). The ACC systems may include longitudinal ACC and/orlateral ACC. Longitudinal ACC monitors and controls the distance to thevehicle immediately ahead of the vehicle 102 and automatically adjustthe vehicle speed to maintain a safe distance from vehicles ahead.Lateral ACC performs distance keeping, and advises the vehicle 102 tochange lanes when necessary. Lateral ACC is related to other ADASapplications such as LCA and CWS.

CACC uses information from other vehicles that may be received via thenetwork interface 2548 and/or the wireless antenna(s) 2546 from othervehicles via a wireless link, or indirectly, over a network connection(e.g., over the Internet). Direct links may be provided by avehicle-to-vehicle (V2V) communication link, while indirect links may beinfrastructure-to-vehicle (I2V) communication link. In general, the V2Vcommunication concept provides information about the immediatelypreceding vehicles (e.g., vehicles immediately ahead of and in the samelane as the autonomous vehicle 1502), while the I2V communicationconcept provides information about traffic further ahead. CACC systemsmay include either or both I2V and V2V information sources. Given theinformation of the vehicles ahead of the autonomous vehicle 1502, CACCmay be more reliable and it has potential to improve traffic flowsmoothness and reduce congestion on the road.

FCW systems are designed to alert the driver to a hazard, so that thedriver may take corrective action. FCW systems use a front-facing cameraand/or RADAR sensor(s) 2528, coupled to a dedicated processor, DSP,FPGA, and/or ASIC, that is electrically coupled to driver feedback, suchas a display, speaker, and/or vibrating component. FCW systems mayprovide a warning, such as in the form of a sound, visual warning,vibration and/or a quick brake pulse.

AEB systems detect an impending forward collision with another vehicleor other object, and may automatically apply the brakes if the driverdoes not take corrective action within a specified time or distanceparameter. AEB systems may use front-facing camera(s) and/or RADARsensor(s) 2528, coupled to a dedicated processor, DSP, FPGA, and/orASIC. When the AEB system detects a hazard, it typically first alertsthe driver to take corrective action to avoid the collision and, if thedriver does not take corrective action, the AEB system may automaticallyapply the brakes in an effort to prevent, or at least mitigate, theimpact of the predicted collision. AEB systems, may include techniquessuch as dynamic brake support and/or crash imminent braking.

LDW systems provide visual, audible, and/or tactile warnings, such assteering wheel or seat vibrations, to alert the driver when theautonomous vehicle 1502 crosses lane markings. An LDW system does notactivate when the driver indicates an intentional lane departure, byactivating a turn signal. LDW systems may use front-side facing cameras,coupled to a dedicated processor, DSP, FPGA, and/or ASIC, that iselectrically coupled to driver feedback, such as a display, speaker,and/or vibrating component.

LKA systems are a variation of LDW systems. LKA systems provide steeringinput or braking to correct the autonomous vehicle 1502 if theautonomous vehicle 1502 starts to exit the lane.

BSW systems detects and warn the driver of vehicles in an automobile'sblind spot. BSW systems may provide a visual, audible, and/or tactilealert to indicate that merging or changing lanes is unsafe. The systemmay provide an additional warning when the driver uses a turn signal.BSW systems may use rear-side facing camera(s) and/or RADAR sensor(s)2528, coupled to a dedicated processor, DSP, FPGA, and/or ASIC, that iselectrically coupled to driver feedback, such as a display, speaker,and/or vibrating component.

RCTW systems may provide visual, audible, and/or tactile notificationwhen an object is detected outside the rear-camera range when theautonomous vehicle 1502 is backing up. Some RCTW systems include AEB toensure that the vehicle brakes are applied to avoid a crash. RCTWsystems may use one or more rear-facing RADAR sensor(s) 2528, coupled toa dedicated processor, DSP, FPGA, and/or ASIC, that is electricallycoupled to driver feedback, such as a display, speaker, and/or vibratingcomponent.

Conventional ADAS systems may be prone to false positive results whichmay be annoying and distracting to a driver, but typically are notcatastrophic, because the ADAS systems alert the driver and allow thedriver to decide whether a safety condition truly exists and actaccordingly. However, in an autonomous vehicle 1502, the autonomousvehicle 1502 itself must, in the case of conflicting results, decidewhether to heed the result from a primary computer or a secondarycomputer (e.g., a first controller or a second controller). For example,in some embodiments, the ADAS system 2706 may be a backup and/orsecondary computer for providing perception information to a backupcomputer rationality module. The backup computer rationality monitor mayrun a redundant diverse software on hardware components to detect faultsin perception and dynamic driving tasks. Outputs from the ADAS system2706 may be provided to a supervisory MCU. If outputs from the primarycomputer and the secondary computer conflict, the supervisory MCU mustdetermine how to reconcile the conflict to ensure safe operation.

In some examples, the primary computer may be configured to provide thesupervisory MCU with a confidence score, indicating the primarycomputer's confidence in the chosen result. If the confidence scoreexceeds a threshold, the supervisory MCU may follow the primarycomputer's direction, regardless of whether the secondary computerprovides a conflicting or inconsistent result. Where the confidencescore does not meet the threshold, and where the primary and secondarycomputer indicate different results (e.g., the conflict), thesupervisory MCU may arbitrate between the computers to determine theappropriate outcome.

The supervisory MCU may be configured to run a neural network(s) that istrained and configured to determine, based on outputs from the primarycomputer and the secondary computer, conditions under which thesecondary computer provides false alarms. Thus, the neural network(s) inthe supervisory MCU may learn when the secondary computer's output maybe trusted, and when it cannot. For example, when the secondary computeris a RADAR-based FCW system, a neural network(s) in the supervisory MCUmay learn when the FCW system is identifying metallic objects that arenot, in fact, hazards, such as a drainage grate or manhole cover thattriggers an alarm. Similarly, when the secondary computer is acamera-based LDW system, a neural network in the supervisory MCU maylearn to override the LDW when bicyclists or pedestrians are present anda lane departure is, in fact, the safest maneuver. In embodiments thatinclude a neural network(s) running on the supervisory MCU, thesupervisory MCU may include at least one of a DLA or GPU suitable forrunning the neural network(s) with associated memory. In preferredembodiments, the supervisory MCU may comprise and/or be included as acomponent of the SoC(s) 2702.

In other examples, ADAS system 2706 may include a secondary computerthat performs ADAS functionality using traditional rules of computervision. As such, the secondary computer may use classic computer visionrules (if-then), and the presence of a neural network(s) in thesupervisory MCU may improve reliability, safety and performance. Forexample, the diverse implementation and intentional non-identity makesthe overall system more fault-tolerant, especially to faults caused bysoftware (or software-hardware interface) functionality. For example, ifthere is a software bug or error in the software running on the primarycomputer, and the non-identical software code running on the secondarycomputer provides the same overall result, the supervisory MCU may havegreater confidence that the overall result is correct, and the bug insoftware or hardware on primary computer is not causing material error.

In some examples, the output of the ADAS system 2706 may be fed into theprimary computer's perception block and/or the primary computer'sdynamic driving task block. For example, if the ADAS system 2706indicates a forward crash warning due to an object immediately ahead,the perception block may use this information when identifying objects.In other examples, the secondary computer may have its own neuralnetwork which is trained and thus reduces the risk of false positives,as described herein.

The autonomous vehicle 1502 may further include the infotainment SoC2704 (e.g., an in-vehicle infotainment system (IVI)). Althoughillustrated and described as a SoC, the infotainment system may not be aSoC, and may include two or more discrete components. The infotainmentSoC 2704 may include a combination of hardware and software that may beused to provide audio (e.g., music, a personal digital assistant,navigational instructions, news, radio, etc.), video (e.g., TV, movies,streaming, etc.), phone (e.g., hands-free calling), network connectivity(e.g., LTE, Wi-Fi, etc.), and/or information services (e.g., navigationsystems, rear-parking assistance, a radio data system, vehicle relatedinformation such as fuel level, total distance covered, brake fuellevel, oil level, door open/close, air filter information, etc.) to theautonomous vehicle 1502. For example, the infotainment SoC 2704 mayradios, disk players, navigation systems, video players, USB andBluetooth connectivity, carputers, in-car entertainment, Wi-Fi, steeringwheel audio controls, hands free voice control, a heads-up display(HUD), an HMI display 2522, a telematics device, a control panel (e.g.,for controlling and/or interacting with various components, features,and/or systems), and/or other components. The infotainment SoC 2704 mayfurther be used to provide information (e.g., visual and/or audible) toa user(s) of the vehicle, such as information from the ADAS system 2706,autonomous driving information such as planned vehicle maneuvers,trajectories, surrounding environment information (e.g., intersectioninformation, vehicle information, road information, etc.), and/or otherinformation.

The infotainment SoC 2704 may include GPU functionality. Theinfotainment SoC 2704 may communicate over the bus 2728 (e.g., CAN bus,Ethernet, etc.) with other devices, systems, and/or components of theautonomous vehicle 1502. In some examples, the infotainment SoC 2704 maybe coupled to a supervisory MCU such that the GPU of the infotainmentsystem may perform some self-driving functions in the event that theprimary controller(s) 2544 (e.g., the primary and/or backup computers ofthe autonomous vehicle 1502) fail. In such an example, the infotainmentSoC 2704 may put the autonomous vehicle 1502 into a chauffeur to safestop mode, as described herein.

The autonomous vehicle 1502 may further include an instrument cluster2520 (e.g., a digital dash, an electronic instrument cluster, a digitalinstrument panel, etc.). The instrument cluster 2520 may include acontroller and/or supercomputer (e.g., a discrete controller orsupercomputer). The instrument cluster 2520 may include a set ofinstrumentation such as a speedometer, fuel level, oil pressure,tachometer, odometer, turn indicators, gearshift position indicator,seat belt warning light(s), parking-brake warning light(s),engine-malfunction light(s), airbag (SRS) system information, lightingcontrols, safety system controls, navigation information, etc. In someexamples, information may be displayed and/or shared among theinfotainment SoC 2704 and the instrument cluster 2520. In other words,the instrument cluster 2520 may be included as part of the infotainmentSoC 2704, or vice versa.

FIG. 28 is a system 2800 diagram for communication between cloud-basedserver(s) and the example autonomous vehicle 1502 of FIG. 25 , inaccordance with some embodiments of the present disclosure. The system2800 may include server(s) 2802, network(s) 1504, and vehicles,including the autonomous vehicle 1502. The server(s) 2802 may include aplurality of GPUs 2808, PCIe switches 2812, and/or CPUs 2816. The GPUs2808, the CPUs 2816, and the PCIe switches 2812 may be interconnectedwith high-speed interconnects such as, for example and withoutlimitation, NVLink interfaces 2810 developed by NVIDIA and/or PCIeconnections 2814. In some examples, the GPUs 2808 are connected viaNVLink and/or NVSwitch SoC and the GPUs 2808 and the PCIe switches 2812are connected via PCIe interconnects. Although eight GPUs 2808, two CPUs2816, and two PCIe switches 2812 are illustrated, this is not intendedto be limiting. Depending on the embodiment, each of the server(s) 2802may include any number of GPUs 2808, CPUs 2816, and/or PCIe switches2812. For example, the server(s) 2802 may each include eight, sixteen,thirty-two, and/or more GPUs 2808.

The server(s) 2802 may receive, over the network(s) 1504 and from thevehicles, image data representative of images showing unexpected orchanged road conditions, such as recently commenced road-work. Theserver(s) 2802 may transmit, over the network(s) 1504 and to thevehicles, neural networks 2806, updated neural networks 2806, and/or2804, including information regarding traffic and road conditions. Theupdates to the 2804 may include updates for the HD map 2708, such asinformation regarding construction sites, potholes, detours, flooding,and/or other obstructions. In some examples, the neural networks 2806,the updated neural networks 2806, and/or the 2804 may have resulted fromnew training and/or experiences represented in data received from anynumber of vehicles in the environment, and/or based on trainingperformed at a datacenter (e.g., using the server(s) 2802 and/or otherservers).

The server(s) 2802 may be used to train machine learning models (e.g.,neural networks) based on training data. The training data may begenerated by the vehicles, and/or may be generated in a simulation(e.g., using a game engine). In some examples, the training data istagged (e.g., where the neural network benefits from supervisedlearning) and/or undergoes other pre-processing, while in other examplesthe training data is not tagged and/or pre-processed (e.g., where theneural network does not require supervised learning). Once the machinelearning models are trained, the machine learning models may be used bythe vehicles (e.g., transmitted to the vehicles over the network(s)1504, and/or the machine learning models may be used by the server(s)2802 to remotely monitor the vehicles.

In some examples, the server(s) 2802 may receive data from the vehiclesand apply the data to up-to-date real-time neural networks for real-timeintelligent inferencing. The server(s) 2802 may include deep-learningsupercomputers and/or dedicated AI computers powered by GPUs 2808, suchas a DGX and DGX Station machines developed by NVIDIA. However, in someexamples, the server(s) 2802 may include deep learning infrastructurethat use only CPU-powered datacenters.

The deep-learning infrastructure of the server(s) 2802 may be capable offast, real-time inferencing, and may use that capability to evaluate andverify the health of the processors, software, and/or associatedhardware in the autonomous vehicle 1502. For example, the deep-learninginfrastructure may receive periodic updates from the autonomous vehicle1502, such as a sequence of images and/or objects that the autonomousvehicle 1502 has located in that sequence of images (e.g., via computervision and/or other machine learning object classification techniques).The deep-learning infrastructure may run its own neural network toidentify the objects and compare them with the objects identified by theautonomous vehicle 1502 and, if the results do not match and theinfrastructure concludes that the AI in the autonomous vehicle 1502 ismalfunctioning, the server(s) 2802 may transmit a signal to theautonomous vehicle 1502 instructing a fail-safe computer of theautonomous vehicle 1502 to assume control, notify the passengers, andcomplete a safe parking maneuver.

For inferencing, the server(s) 2802 may include the GPUs 2808 and one ormore programmable inference accelerators (e.g., NVIDIA's TensorRT 3).The combination of GPU-powered servers and inference acceleration maymake real-time responsiveness possible. In other examples, such as whereperformance is less critical, servers powered by CPUs, FPGAs, and otherprocessors may be used for inferencing.

“Logic” is used herein to machine memory circuits, non transitorymachine readable media, and/or circuitry which by way of its materialand/or material-energy configuration comprises control and/or proceduralsignals, and/or settings and values (such as resistance, impedance,capacitance, inductance, current/voltage ratings, etc.), that may beapplied to influence the operation of a device. Magnetic media,electronic circuits, electrical and optical memory (both volatile andnonvolatile), and firmware are examples of logic. Logic specificallyexcludes pure signals or software per se (however does not excludemachine memories comprising software and thereby forming configurationsof matter).

Various functional operations described herein may be implemented inlogic that is referred to using a noun or noun phrase reflecting saidoperation or function. For example, an association operation may becarried out by an “associator” or “correlator”. Likewise, switching maybe carried out by a “switch”, selection by a “selector”, and so on.

Within this disclosure, different entities (which may variously bereferred to as “units,” “circuits,” other components, etc.) may bedescribed or claimed as “configured” to perform one or more tasks oroperations. This formulation—[entity] configured to [perform one or moretasks]—is used herein to refer to structure (i.e., something physical,such as an electronic circuit). More specifically, this formulation isused to indicate that this structure is arranged to perform the one ormore tasks during operation. A structure can be said to be “configuredto” perform some task even if the structure is not currently beingoperated. A “credit distribution circuit configured to distributecredits to a plurality of processor cores” is intended to cover, forexample, an integrated circuit that has circuitry that performs thisfunction during operation, even if the integrated circuit in question isnot currently being used (e.g., a power supply is not connected to it).Thus, an entity described or recited as “configured to” perform sometask refers to something physical, such as a device, circuit, memorystoring program instructions executable to implement the task, etc. Thisphrase is not used herein to refer to something intangible.

The term “configured to” is not intended to mean “configurable to.” Anunprogrammed FPGA, for example, would not be considered to be“configured to” perform some specific function, although it may be“configurable to” perform that function after programming.

Reciting in the appended claims that a structure is “configured to”perform one or more tasks is expressly intended not to invoke 35 U.S.C.§ 112(f) for that claim element. Accordingly, claims in this applicationthat do not otherwise include the “means for” [performing a function]construct should not be interpreted under 35 U.S.C § 112(f).

As used herein, the term “based on” is used to describe one or morefactors that affect a determination. This term does not foreclose thepossibility that additional factors may affect the determination. Thatis, a determination may be solely based on specified factors or based onthe specified factors as well as other, unspecified factors. Considerthe phrase “determine A based on B.” This phrase specifies that B is afactor that is used to determine A or that affects the determination ofA. This phrase does not foreclose that the determination of A may alsobe based on some other factor, such as C. This phrase is also intendedto cover an embodiment in which A is determined based solely on B. Asused herein, the phrase “based on” is synonymous with the phrase “basedat least in part on.”

As used herein, the phrase “in response to” describes one or morefactors that trigger an effect. This phrase does not foreclose thepossibility that additional factors may affect or otherwise trigger theeffect. That is, an effect may be solely in response to those factors,or may be in response to the specified factors as well as other,unspecified factors. Consider the phrase “perform A in response to B.”This phrase specifies that B is a factor that triggers the performanceof A. This phrase does not foreclose that performing A may also be inresponse to some other factor, such as C. This phrase is also intendedto cover an embodiment in which A is performed solely in response to B.

As used herein, the terms “first,” “second,” etc. are used as labels fornouns that they precede, and do not imply any type of ordering (e.g.,spatial, temporal, logical, etc.), unless stated otherwise. For example,in a register file having eight registers, the terms “first register”and “second register” can be used to refer to any two of the eightregisters, and not, for example, just logical registers 0 and 1.

When used in the claims, the term “or” is used as an inclusive or andnot as an exclusive or. For example, the phrase “at least one of x, y,or z” means any one of x, y, and z, as well as any combination thereof.

Having thus described illustrative embodiments in detail, it will beapparent that modifications and variations are possible withoutdeparting from the scope of the invention as claimed. The scope ofinventive subject matter is not limited to the depicted embodiments butis rather set forth in the following Claims.

1. A method for modifying the behavioral logic of an autonomous vehicle,the method comprising: assigning a narrowness metric to each of one ormore paths in a driving scenario, a path comprising a sequence ofdriving states of the autonomous vehicle; wherein the narrowness metricfor a particular state in a path is determined based on a number ofchild states of the articular state; assigning an effort metric to thepaths; assigning an amount of time for action by the autonomous vehicleto avoid collision along the paths; and updating the behavioral logic ofthe autonomous vehicle based on a scoring of the paths.
 2. The method ofclaim 1, wherein each node of the paths is assigned a node effortmetric.
 3. The method of claim 2, wherein: the effort metric of a pathis a sum of the node effort metrics for the path; and the node effortmetric is determined as a sum of steering and acceleration values toreach the node from a parent node of the path.
 4. The method of claim 1,wherein the paths are generated by introducing perturbations into thedriving scenario.
 5. A non-transitory computer-readable storage medium,the computer-readable storage medium including instructions that whenexecuted by a computer, cause the computer to: characterize a path in adriving scenario according to narrowness and effort of the path; whereinthe narrowness of the path is determined b a number of children at eachnode of the path; assign to each node of the path a minimum amount oftime for action by the vehicle to avoid collision from the node; andsimulate the path in the computer.
 6. The computer-readable storagemedium of claim 5, the computer-readable storage medium includinginstructions that when executed by a computer, cause the computer to:score the path based at least in part on the effort of the path. 7.(canceled)
 8. A method for computer simulation of a driving scenario foran autonomous vehicle, the method comprising: configuring the drivingscenario with a set of road descriptors on which a set of actorsoperate, each actor assigned a behavioral policy, and wherein at leastone actor is the autonomous vehicle; deriving a multivariate metric fora number of safe driving paths in the driving scenario; assigning anarrowness metric to each of the paths in the driving scenario, whereinthe narrowness metric is determined based on a number of child nodes ofeach node of the path; assigning an effort metric to nodes of the paths;and assigning to each node a minimum amount of time for action by theautonomous vehicle to avoid collision from the node; executing thedriving scenario in the computer simulation; and improving a behavioralpolicy of the autonomous vehicle based on results of the computersimulation.
 9. The method of claim 8 further comprising: generatingunsafe paths into the driving scenario by introducing perturbations intothe driving scenario.
 10. The method of claim 8 further comprising:scoring the driving scenario based on performance of the autonomousvehicle in the computer simulation; prioritizing the driving scenariobased on the scoring; and applying the prioritization to improve thebehavioral policy of the autonomous vehicle.
 11. (canceled)
 12. Themethod of claim 8 wherein the effort for a path is computed as a sum ofnode efforts in the path.
 13. The method of claim 8 wherein themultivariate metric comprises: a total number of paths to evaluate froma beginning of the driving scenario to an end of the driving scenario,wherein the total number of paths forms a tree comprising branches; anumber of no-collision driving paths from the beginning of the drivingscenario to the end of the driving scenario; an average effort for theno-collision driving paths; a minimum no-collision driving path effort;a minimum amount of time before a collision occurs in a branch of thetree comprising at least one no-collision driving path; and an averageof a minimum number of branches of the tree along each no-collisiondriving path.
 14. The method of claim 8 wherein: a rate of change of astate of an actor is defined as a control policy represented by$f = {\frac{ds}{dt} = \left\lbrack {a\frac{\upsilon\tan(\delta)}{L}} \right\rbrack}$wherein a is an acceleration of the actor, δ is a steering angle of theactor, L is a wheelbase of the actor, and v is a scalar velocity of theactor.
 15. The method of claim 8 further comprising: identifying acontrol policy for an unsafe actor A; applying the control policy forthe unsafe actor to generate a trajectory that minimizes an objectivefunction γ represented by γ_(AE)=D² _(AE)=∥x_(A)−x_(E)∥₂²=(x_(iA)−x_(iE))²+(x_(jA)−x_(jE))² wherein a position of actors A and Ein the driving scenario is defined by a vector x=[x_(i) x_(j)] ofCartesian coordinates and wherein a square of a Euclidean distancebetween the unsafe actor and an autonomous vehicle E is minimized. 16.The method of claim 15, wherein: reducing the objective function withtime based on a constraint on a derivative of γAE such that$\frac{\partial{\gamma}_{AE}}{\partial t} < 0.$
 17. The method of claim16, wherein the constraint is applied such that the autonomous vehicledoes not react to the unsafe actor and therefore the control policy ofthe autonomous vehicle is fixed.
 18. The method of claim 17 wherein theconstraint is relaxed on condition that the autonomous vehicle appliesSafety Force Field (SFF).
 19. The method of claim 8 wherein thebehavioral policy for an actor is represented as: β₁a_(A)+β₂tan(δ_(A))<const. where a is acceleration of the actor, and δ is asteering angle for the actor.
 20. The method of claim 19 whereinadditional constraints are placed on a maximum value of the steeringangle and the acceleration.
 21. The method of claim 8 further comprisingthe application of a plurality of modes, wherein: a first mode forselecting a control policy for an unsafe actor comprises a policy withmaximum acceleration and maximum steering; a second mode for selectingthe control policy for the unsafe actor comprises a policy with maximumacceleration and minimum steering; and a third mode for selecting thecontrol policy for the unsafe actor comprises a policy with maximumsteering and minimum acceleration.
 22. The method of claim 21 whereinthe first mode comprises a maximum effort for creating an unsafe drivingscenario.
 23. The method of claim 21 further comprising a fourth modefor selecting the control policy for the unsafe actor, the fourth modecomprising a minimum effort for creating an unsafe driving scenario.